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ABSTRACT 

An evaluation was made of the biomedical Computer 
Assisted Instruction (CAI) Network Experiment, established by the 
National Library of Medicine in 1973 to test the feasibility of 
sharing CAI learning materials through a national computer network. 
The evaluation was designed to assist decision makers in planning a 
future mechanism for distributing biomedical CAI. The study focused 
on the institutions (95 as of October 1974) and people who use the 
network and the contractors who supplied the CAI library. Data 
sources included case studies, user reports, and interviews with 
hospital and medical school administrators, faculty, librarians, 
computer laboratory staff, and students. Network usage by program 
type, by user institution, and by class of user were analyzed. Also 
analyzed were the impact of the network on curricula, faculty, 
students, libraries, and media centers as well as the cost /benefits. 
The report includes a roster of the participants in the network 
experiment. (Author/EMH) 
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PREFACE 

This report describes an evaluation, 'made by the Human Resources Research 
Organization for the National Library of Medicine, of the Lister Hill Center Experimental 
Computer-Assisted Instruction Network, a biomedical communications network. Data for 
the evaluation were obtained from several sources, including interviews with adminis- 
trators, faculty, library personnel, and students; Network user reports; and meetings of 
the Health Education Network User's Group. 

The Network Experiment provided a wealth of experience for educators, students, 
and technologists among many dimensions, including quality of computer-assisted instruc- 
tion (CAI) materials, distribution of the libraries, the costs of networking as a distribu- 
tion mechanism, and institutional variables affecting CAI use. 

The work for this project was performed by HumRRO's Eastern Division, 
Alexandria, Virginia, Dr. J. Daniel Lyons, Division Director. Dr. Robert J. Seidel was the 
Principal Investigator, and Martin L. Rubin was the Project Director. Other members of 
the research staff were Beverly Hunter, Marilyn Knetsch, and Rosalake Coleman. 

The research upon which this publication is based was performed pursuant to 
Contract No. NOl-LM-4-4725 with the National Library of Medicine, National Institutes of 
Health, Department of Health, Education and Welfare. 
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EXECUTIVE SUMMARY 



The Lister Hill Center Experimental CAI Network (referred to in this report as the 
**Network Experiment'') was established in July 1972. Its purpose was to test the 
feasibility of sharing CAI learning materials through a national computer network. 

The Network Experiment was not intended to establish an operational distribution 
mechanism for CAI libraries, but rather, to provide an experiential setting for evaluating 
the new medium and the problems and prospects for national distribution. Formal 
support from the Nation al'^Library of Medicine (NLM) for the Network is programmed to 
terminate in May 1975. 

The Human Resources Research Organization (HumRRO) conducted an evaluation 
of the Network which was designed to assist decision makers in planning future 
mechanisms for distributing biomedical CAI, and to identify additional research and 
development needs. 

The focus of the Network evaluation study was on the institutions and people who 
use the Network. Information was gathered from: 

(1) Field trips to selected user and contractor institutions to interview 
administrators, faculty, library and media center personnel, and students, 

(2) Bimonthly reports to the Lister Hill Center from user institutions. These 
reports address a wide range of technical, educational, financial, and administrative 
matters which were encountered in the Network Experiment. 

(3) Data from contractors on CAI program usage by user institutions. 

(4) Meetings of the Health Education Network Users' Group (HENUG) of the 
Association for Development of Cpmputer-Based Instructional Systems (ADCIS). 



USER INSTITUTIONS 

A total of 95 institutions had participated at some time in the Network Experiment 
as of October 1, 1974. These institutions included medical schools, teaching hospitals, 
research institutes and associations, health science centers, continuing education centers, 
schools of dentistry, and schools of nursing. Administrative support at user institutions 
was lacking. Medical school administrators were impressed with the educational utility of 
CAI but were not convinced that the add-on costs could be justified. They wanted proof 
of the learning benefits of CAI. The positive evidence that surfaced during the 
study— highly favorable opinions, surprisingly high levels of usage, willingness to bear part 
of the Network costs— was not sufficient. 

Participating institutions varied widely in the extent to which they used the 
Network. By August 1974, Network users had logged a total of 11,090 hours on the 
Ohio State University (OSU) system. Of the 72 institutions that have used the OSU 
system, six institutions accounted for approximately 60% of the usage hours;* twenty of 
the institutions used less than 10 hours each. 

By August 1974, Network users had logged a total of 21,220 hours on the 
Massachusetts General Hospital (MGH) system. Of the 90 institutions that have used the 
system, six accounted for 37% of the total usage hours.^ 

^University of California at Los Angeles (UCLA), University of Washington, George Washington 
University, University of Oregon School of Nursing, University of the Pacific, and Fort Worth Osteopathic 
Hospital. 

^UCLA, University of Pennsylvania, Harvard, Medical College of Virginia, University of Arizona, 
and George Washington University. 
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By February 1974, when the University of Illinois Medical College (UIMC) left the 
Network, users had logged a total of 5,144 hours on the UIMC system. Of the 63 
institutions that used the system, six accounted for 54% of the total usage hours.* 
Twenty-four institutions used the UIMC materials for less than 10 hours each. 

Most institutions made some physical modifications in order to accommodate the 
computer terminals. Study carrels were built in media labs and libraries. Users built or 
designed special facilities to house the computer terminal. Soundproof rooms were 
constructed so that students, working in small groups, would not disturb other library 
users. Stanford University had a unique CAI study area. The terminal was located in the 
Media Center, in front of a comfortable^oucfi where small groups of students could 
easily work together. The terminal was connected to a large video display monitor that 
provided a large screen image of the text. The typical Lister Hill user had one terminal 
but some of the larger users had as many as four terminals. 

The University of Connecticut School of Medicine provided portable terminals in 
addition to having several terminals within its center. In the short time that these 
terminals have been in use, they have become very popular. There have been recent 
requests by physicians to borrow a terminal for trial use in their offices. 

The Network resource was not cost-effective for user institutions. The CAI materials 
provided a supplement to existing curricula. The costs of using the Network could not be 
taken out of existing budgets. 

The user institutions* Network funds came from outside sources, such as grants, 
donations, and special funds. In some cases support came from funds from the dean's 
office but not from the operational departmental budgets. Institutions were required to 
pay for their own terminals. Many schools purchased the terminals since they were a 
relatively small, one-time cost. The schools also were required to pay the local charges for 
linking into the nearest TYMSHARE communications node. If the school was at some 
distance from the nearest node, phone charges for this hook-up could amount to as much 
as $l,000/month. However, if the institution was in a TYMSHARE node city, such as 
Los Angeles, there were no local phone charges. 

Initially, the Network service itself was free to users. It was later decided to have 
Network users pay an increasing portion of the cost. The initial charge was $2.50/hour as 
of February 1974. In July 1974, the charge was raised to $5.00/hour. Since total 
Network costs range between $16 and $20 an hour, depending on total usage. Lister Hill 
is still subsidizing the major portion of the Network cost. Approximately 45% of these 
costs were communications costs; another 25% was the computer cost at the Network 
host facilities. Personnel staff at the host sites and at the Lister Hill Center made up the 
cost balance. 

CAI LIBRARY USE 

The CAI materials were used as a supplementary learning resource. While the CAI 
leammg materials were perceived by both faculty and students to be a valuable 
educational addition, as measured by student usage and opinion surveys, they did not 



UCLA, University of Pennsylvania, Medical College of Virginia, University of Texas at San Antonio 
and Galveston, and Stanford University. 
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bring about any major changes in course objectives or content. The CAI materials served 
mainly to enrich, or supplement, the regular program. 

Curriculum integration is of paramount importance for long-t erm commitment. 
There were only scattered instances of any attempt to integrate the MGH clinical 
encounter programs into the clerkship curriculum. Even at UCLA, the largest user school, 
there was virtually no faculty interest in becoming involved with CAI. A major reason for 
this lack of enthusiasm at most institutions was that faculty members were hesitant to 
reorganize their courses to include CAI materials because of the planned termination of 
the Network. Without this faculty involvement, the schools will not back large budgetary 
expenditures for CAI. " 

The OSU materials, because of their tutorial nature, require some form of cur- 
riculum integration. This may explain why OSU's program usage has been far less than 

MGH's program usage. 

Several user institutions stated that the principal reason for termmatmg their partici- 
pation in the CAI Network was that they could not justify the Network charges if the 
materials were to be only an appendage to the regular curriculum. 

Institutions with a traditional lecture-oriented, lock-step curriculum reported 
difficulty in scheduling CAI usage. The computer tutorials were in no instance used as 
final examinations. Therefore, the utilization of CAI did. not have any effect on 
established procedures for grading. Many of the user institutions had previously shifted 
fiom a high lecture orientation to a high emphasis on self-directed study. These mstitu- 
tions had excellent facilities for self-directed learning. They were anxious to provide still 
another modality in addition to video cassettes, audio tapes, heart sound synthesizers, 
and other devices. A recurring problem in the experimental Network was the lack of a 
constituted '..^dy to judge the merits of CAI programs. Network hosts were placed in the 
position of having to determine the quality of their own programs. 

A major strength of the Network approach to CAI distribution was the facility it 
p rovided for improving and revising curricular materials . The user institutions played an 
indispensable role in providing feedback to the authors concerning errors in the medical 
content; program logic problems, deficiencies in explanation, and alternative treatments 
and diagnoses. MGH received literally thousands of individual comments from interested 
users who expressed a personal interest in correcting perceived weaknesses of the CAI 
programs. In as many cases as possible the user received a personal reply, and when the 
author was in agreement the programs were revised. 

Recommendations 



It would be of significant benefit to involve the various speciality boards and 
associations in a review function. The Association of American Medical Colleges would be 
the logical organization to develop the overall objectives, goals, and procedures for 
establishing CAI program review mechanisms. 

Similariy, the Lister Hill Experiment indicates the inadequacy of the scope of the 
offered CAI programs. A survey of the educational needs of various user populations 
should be undert;aken to learn those subject areas in which CAI materials are needed. 




CAI MATERIALS ASSESSMENT 



One finding that emerged clearly from the Network Experiment was that the 
benefits of CAI distribution accrued directly to students, rather than to the institutions 
or faculty. Students benefitted in a number of ways, receiving: 

• Remedial learning. 

• Counseling. 

• Review for examinations. 

• Problem-solving experiences. 

• Computer awareness they can apply later as practicing physicians. . 

• Wide range of simulated clinical experience. 

Students in medical schools and other health professions used Network CAI 
materials an average of 1800 hours a month. Usually the students used materials on their 
own initiative, without faculty direction. The great majority of Network materials were 
used by medical students in their clerkship and pre-clerkship years. 

No provision was made in the Network Experiment for a systematic assessment 
of the impact of the materials on student learning. Each user institution was expected to 
provide its own evaluation of the materials and their use. However, only a few insti- 
tutions were able to complete any kind of systematic study. Furthermore, each insti- 
tution that performed some kind of study performed it in its own way. Objective 
summary data regarding students learning are available for only a few institutional 
evaluations. A number of problems prevented most of the studies from being completed: 
lack of faculty member time to spend on educational research, legal and technical 
problems in using the Network (especially in the first year of the Network Experiment), 
and priorities given to medical research. 

Recommendation 



The Lister Hill Network Experiment indicates a strong need for more substantial 
evidence of CAI program benefits, including inKiepth studies of learning benefits. The 
Library, therefore, should sponsor experiments by CAI user institutions which measure 
changes in student performance directly attributable to CAI. 



FACULTY ATTITUDES 

The Network had "much less impact on faculty than on students. Faculty members 
who attended demonstrations of the Network Experiment had a generally favorable 
reaction to the materials. Most faculty endorsed the idea of using the resource but left it 
up to the students as to how they would use it. 

As a condition for receiving use of the Network facilities, user institutions were 
asked by the Lister Hill Center to prepare CAI materials to be added to the library. 
Medical school faculty members were expected to develop these materials. In fact, it was 
hoped that they would become involved with the total process. However, faculty (with 
some exceptions) did not develop such materials or use them in their courses. The 
Network Experiment did not provide funds for faculty release time to perform these 
development activities. Authoring course materials or devising new computer-oriented 
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curricula is extremely time consuming and, in addition, requires specialized computer 
skills. Therefore, the lack of faculty involvement was not surprising. 

In the final year of the Network Experiment, faculty members are beginning to take 
a more active role, with over 50 instructional units currently being developed by faculty 
al user institutions. 

Schools with a full-time faculty devoted to teaching and research (and without 
outside private practice) were more amenable to committing time to exploring CAL At 
the University of Washington, which has a full-time faculty, there are as many as 25 
faculty members working on and developing CAI modules, including the extensive 
reworking of the visual materials that accompany CAI programs. This involvement 
contrasts with UCLA faculty members who have heavy involvements in private practice 
and thus participated very little in the use of the Lister Hill Network. The faculty 
committee at UCLA voted agamst continuing the project after termination of Lister 
Hill support. 

Recommendation 

Faculty should be given a more prominent role in future Lister Hill CAI programs. 

POSSIBLE ALTERNATIVES TO" NETWORK 

Other methods^^ol CAI distribution may provide less expensive distribution service 
(e.gM physical trilnspor^.of programs). This would not, however, provide the built-in 
compatibility, feedback, and revision process of a hardwired network. Alternatives such as 
a clearinghouse, online bibliographic service, intelligent terminals, and desktop computers 
should be explored. 

Recommendation 

Lister Hill Center should sponsor a research program to 'evaluate low-cost distri- 
bution mechanisms, such as a Network of microprocessors* intelligent terminals, or 
a clearinghouse. 
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Chapter 1 
INTRODUCTION 



PURPOSE OF THE STUDY 

This report, undertaken by the Human Resources Research Organization in May 
1974, fummarizes the findings of the evaluation of the Lister Hill Center Experimental 
Computer-Assisted Instruction (CAT) Network (referred to in this report as the "Network 
Experiment''). The purposes of the study were to describe and analyze the impact and 
experience of the Network Experiment in ways that would assist decision makers in 
planning future mechanisms for distributing biomedical CAI, and to identify additional 
research and development needs. 

This report is addressed not only to the sponsors and participants of the Network 
Experiment, but also to educators and technologists who have an interest in sharing 
learning resources throu^ technology-based distribution systems. Much of what was 
learned in the biomedical Network Experiment has general applicability to other educa- 
tional networking efforts and attempts to share CAI materials. 



DESCRIPTION OF THE NETWORK 

,The Lister Hill Center Experimental CAI Network was established in July 1972. 
Its purpose was to test the feasibility of sharing CAI learning materials through a national 
computer network. 

Soon after the establishment of the Lister Hill Center, Dr. Martin M. Cummings, 
Director of the National Library of Medicine, asked the American Association of Medical 
Colleges (AAMC) to take a leadership position in involving the academic medical 
community in Network plans. A conference was held in February 1969 to consider the 
educational services that a network might provide (Smythe, 1969). Subsequent to this 
conference, the AAMC was asked to recommend more specific plans, which resulted in 
the production of a report from the Steering Committee, Council of Academic Societies, 
Association of American Medical Colleges (Stead, eta/., 1971). This report covered the 
state-of-the-art of instructional technology, the need for a biomedical community net- 
work, factors governing selection of materials, organization and administration, evalua- 
tion, and staff. The Steering Committee report included many recommendations, one of 
which is covered in the following statement: 

"The Steering Committee advocates the organization of a biomedical ccm- 
munications network designed to meet some of the needs of medical education ?^*'J 
medical practice and to capitalize on the current state of development of various phd.^.. > 
of communications and computer technology. Of primary importance is the requireme^^ 
to maintain a high level of learning experiences for growing numbers of students to 
whom medical, dental, nursing and other health career schools are committed.'' 

The AAMC report was presented to members of the Board of Regents of t*^-"* 
National Library of Medicine. The Board appointed a Priorities Review Committee to 
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study the report. This committee presented four recommendations which were adopted 
unanimously by the Regents. One of the four recommendations read: 

"The Committee advocates the organization of a biomedical communications 
network fundamentally conceived as providing the mechanism by means of which inter- 
institutional cooperation and sharing of resources will be used to meet some of the needs 
of medical education. Implementation of this goal began in September 1971.'' 

The original Network was based on four contracts with the National Library of 
Medicine (NLM): one with a commercial timesharing service and three with institutional 
centers having CAI expertise. The three institutions are Massachusetts General Hospital, 
the University of Illinois Medical Center, and Ohio State University College of Medicine. 

Course material from Massachusetts General Hospital (MGH) is primarily intended 
for supplemental use by medical students and physicians involved in continuing education 
and contains programs in Abdominal Pain, Coma, and Jaundice, among others. The 
University of Illinois Medical Center (UIMC)^ offered two types of material: (a) a 
computerized random item bank (CRIB) and (b)a computer-aided simulation of the 
clinical encounter (CASE). The programs from Ohio State University College of Medicine 
(OSU) cover a broad spectrum of health problems and were written for many different 
educational levels (Wooster and Lewis, 1973). 

The purpose of the Network Experiment was not to establish an operational 
distribution mechanism for CAI libraries, but rather, to provide an experiential setting for 
evaluating the new medium and the problems and prospects for national distribution. 
Formal NLM support for the Network itself is programmed to terminate in May 1975. 

DESCRIPTION OF THE EVALUATION STUDY 

The focus of the Network Evaluation Study is on the institutions and people who 
use the Network.^ Information was gathered from four main sources: 

(1) Field trips to selected user and contractor institutions to interview 
administrators, faculty, library and media center personnel, and students. 

(2) Bimonthly reports to the Lister Hill Center from user institutions. These 
reports address a wide range of technical, educational, financial, and administrative 
matters which were encountered in the Network Experiment. 

(3) Data from contractors on CAI program usage by user institutions. 

(4) Meetings of the Health Education Network Users' Group (HENUG) of the 
Association for Development of Computer-Based Instructional Systems (ADCIS). 

The information gathered from these sources was interpreted and organized into 
several categories of study. This report is organized around the following categories: 

• Impact of the Network Experiment on Users 

Institutions 
Curricula 
Faculty 
Students 

Libraries and Media Centers 

• Institutional Variables Affecting Utilization of Network 

• Evaluation of CAI Materials 



*The Uniwrwty of Illinois withdrew from the Network Experiment in February 1974. lU CRIB 
•yitem wm not considered to be a cost-effective use of computers. The CASE system, however, was 
transferred to the Ohio State University computer. 

^Principal persons involved in the Network Experiment are listed in Appendix A. 
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• Cost/Benefit Analysis 

• What was Learned from the Network Experience 

Materials 

Development of materials 
Distribution mechanisms 
Evaluation methods 

• Functional Requirements of a CAI Distribution System 

User Tailoring 
Feedback-Revision Process 
Support Services 
User Documentation 
Peer Review 
People Networks 

• Recommendations for Future Distribution Mechanisms 
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Chapter 2 

IMPACT OF THE NETWORK EXPERIMENT 



Qualitative and quantitative indicators of the impact of the Network Experiment on 
users are organized into five categories: 

• Institutions 

• Curricula 

• Faculty 

• Students 

• Libraries and Media Centers 



IMPACT ON INSTITUTIONS 

A total of 95 institutions had participated at some time in the Network Experiment 
as of October 1, 1974. A breakdown by type of institution follows: 



Medical Schools 


65 


Teaching Hospitals 


9 


Research Institutes and 




Associations 


7 


Health Science Centers 


2 


Continuing Education Centers 


5 


Schools of Dentistry 


2 


Schools of Nursing 


2 


Other 


2 


Total 


95 



Fifty institutions have participated in the Network Experiment for a year or longer; 
twenty-one institutions participated for three months or less, and then dropped out. As 
of December 1974, there were approximately 65 active institutional Network users. 

At the beginning of the Network Experiment, users were designated **trial" or 
"operational," with operational users having greater responsibility to contribute evaluative 
data. It was also anticipated that operational users would contribute to the network CAI 
library. This designation was dropped in February 1974 when usage charges 
were initiated. 

Dr. Harold Wooster and Ms. Jinnet Lewis of the Lister Hill Center conducted a 
statistical analysis of the significance of the presence of an Office of Medical Education 
on the incidence of institutional participation in the Network Experiment (Wooster and 
LewiSy 1974). Using a chi-square analysis, they found that institutions that have an Office 
of Medical Education were much more likely to participate in the Network Experiment. 
Of the three components of a medical institution (i.e., education, research, and service), 
it is presumed that the presence of an Office of Medical Education is an indication of the 
degree of emphasis placed on the educational component. It also means that there is an 
individual with the lime and interest to help investigate and nurture a new learn- 
ing modality. 




The participating institutions varied widely in the extent to which they used the 
Network. (See Table 1 for lists of Network users and dates and amount of time used.) 

By August 1974, Network users had logged a total of 11,090 hours on the Ohio 
State University (OSU) system. Of the 72 institutions which have used OSU's system, six 
institutions accounted for approximately 60% of the usage hours: University of California 
at Los Angeles (UCLA), University of Washington, George Washington University, 
University of Oregon School of Nursing, University of the Pacific, and Fort Worth 
Osteopathic Hospital., Twenty of the institutions using Ohio State's program library used 
a total of less than 10 hours each. 

By August 1974, Network users had logged a total of 21,220 hours on the 
Massachusetts General Hospital (MGH) system. Of the 90 institutions which have used 
the system, six of them accounted for 37% of the total usage hours: UCLA, University of 
Pennsylvania, Harvard, Medical College of Virginia, University of Arizona, and George 
Washington University. 

By February 1974, when the University of Illinois Medical College (UIMC) left the 
Network, Network users had logged a total of 5,144 hours on the UIMC system. Of the 
63 institutions which used the system, six of them accounted for 54% of the total us^e 
hours: UCLA, University of Pennsylvania, Medical College of Virginia, University of 
Texas at San Antonio and Galveston, and Stanford University. Twenty-four institutions 
used the UIMC materials for a total of less than 10 hours each. 

♦ Most institutions made some modifications to the physical plant in order to 
accommodate the computer terminals. Study carrels were built in media labs and 
libraries. Most large institutional users built or designed special facilities to house the 
computer terminal. At UCLA, special soundproof rooms were constructed so that 
students working in small groups would not disturb other library users. Stanford Uni- 
versity had a unique CAI study area. The terminal was located in the Media Center, in 
front of a comfortable couch where small groups of students could easily work together. 
The terminal was connected to a large video display monitor which provided a lai^e 
screen image of the text. The typical Lister HUl user had one terminal but some of the 
larger users had as many as four terminals. 

The University of Connecticut's School of Medicine has adopted the portable 
terminal concept in addition to having several terminals within their center. In the short 
time that these terminals have been in use, they have become very popular. There have 
been recent requests by physicians to borrow a terminal for trial use in their offices.. 

The user institution's Network funds came from outside sources, such as grants, 
donations, and special funds. In some cases support came from funds from the dean's 
office but not from the operational departmental budgets. Institutions were required to 
pay for their own terminals. Many schools purchased the terminals since they were a 
relatively small, one-time cost. The schools also were required to pay the local charges for 
linking into the nearest TYMSHARE communications node. If the school was at some 
distance from the nearest node, phone charges for this hook-up could amount to as much 
as $1 000/month. However, if the institution was in a TYMSHARE node city such as 
Los Aiigeles, there were no local phone charges. A list of TYMSHARE nodes is contained 

in Appendix B. , j x u 

Initially, the Network service itself was free to users. It was later decided to have 
Network users pay an increasing portion of the cost. The initial charge was $2.50/hour as 
of February 1974. In July 1974, the charge was raised to $5.00/hour. Since total 
Network costs range between $16 and $20 an hour, depending on total usage. Lister HUl 
is still subsidizing the major portion of the Network cost. 

While the Network provided an entirely new learning modality to many user 
institutions, it did not affect the overall educational philosophy and program as much as 
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Table 1 

Summary of Network Use 



Usfr 


Usage Period 


Time Ustd 


Mean Hours/ 
Month Usage 


Start 


Continuing^ 

Or FnHinn 

Dateb 


Months^ 


Hours 


Alabama, Univ. of-School of Medicine, 












Birmingham 


11/72 


C 


22 


483.9 


22.0 


Albanv MM^iral CoIImm of Union 












University, Albany, N.Y. 


A 77*3 


u 


1 7 
1 / 


b4D.b 


32.2 


American Academy of Orthopedic 












Surgeons, Chicago, III. 


0/ /4 


7 77>l 


Z 


Q 
.O 


,4 


American College of Radiology, 












Boston, Mass. 


O/ / O 




A 


'3P 9 
oo.z 


19.1 


American Heart Association, 












' Brookline, Mass. 


11/72 


\ 12/72 


2 


5.5 


2.7 


Arizona, Univ. of— Arizona Medical 












Center, Tucson 


lU/ /o 




1 1 


1 VftA 9 


114.9 


Augusta General Hospital, Augusta, Me. 




~" TO 77*3 


\ H 


79 R 
/ Z.9 


5.2 


Baylor Univ., Waco, Tex. 


1 




9 


*^ 1 


1.5 


Beth Israel Hospital, Boston, Mass. 


1/73 


9/73 


9 


93,0 


10.3 


Bird, Kenneth T., M.D., Massachusetts 












General Hospital, Boston 


4/74 


C 


3 


18.4 


6.1 


Boston Univ. Medical Center, University 












Hospital 


1 77*3 

1/ /o 


K/7A 
0/ /H 


1 1 
1 1 


9A R 


2.2 


Brown Univ., Providence, R.I. 


O 77*3 




7 


1R7 9 

1 Q/ .Z 


23.9 


CAPO (Bolt, Beranek, and Newman) 


1 1 no 


19/79 


9 


4 4 


2,2 


California, Univ. of, Davis 


1 1 770 

1 1/ 




9n 


'?9R 1 


16.4 


California, Univ. of, Los Angeles 


11/72 


c 


22 


3,235.1 


147.1 


uaiiTornia, univ. ot— los Mngeies, ueaars- 












Sinai Medical Center 


1 77>l 


p 


Q 
O 


A(\ 1 


5.0 


California, Univ. of-Los Angeles, 












Harbor General Hospital 




p 


Q 
O 


119 0 

1 IZ.U 


14.0 


California, Univ. of, San Diego 


C 77*3 

b//o 




1 ^ 
1 D 


1R9 K 
1 OZ.D 


10.8 


Case Western Reserve Univ., Cleveland, 0. 


< /TO 

1/73 




1 O 
IB 


OOZ.Z 


35.9 


CBX Project, National Board of Medical 












Examiners 


11/72 


10/73 


12 


O 4 o c 

j1o.o 


26.6 


Chicago, Univ. of 


8/73 


rt /TO 

9/73 


o 

Z 




9.9 


Cincinnati Univ. of 


4/74 


6/74 


3 


1.4 


.5 


Columbia Univ., New York-College of 












Physicians and Surgeons, Medical Library 


5/73 


C 


16 


461.8 


28.9 


Columbia Univ., New York-College of 












Physicians and Surgeons, Dept. of 












Pathology 


3/73 


C 


13 


186.1 


14.3 


Connecticut, Univ. of-Health 












Center, Farmington 


5/73 


C 


16 


621.8 


38,9 




(Continued) 
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Table 1 (Continued) 



Summary of Network Use 



Usage Period 


Time Used 


Mean Hours/ 
Month Usage 


Ustr 


Start 


Continuing? 
or Ending 
uate*' 


Months^ 


Hours 


Emergency Medical System Medical 














Group, Palo Alto, Calif. 


9/73 


6/74 


5 


23.6 


4.7 


Florida, Univ. of-J. Hillis Miller Health 














Center, College of Dentistry, 














Gainesville 


6/73 


1/74 


B 


39.1 




4.9 


Fort Worth (Texas) Osteopathic 














Hospital 


4/73 


C 


17 


611.9 




36.0 


George Washington Univ., 














Washington, D.C. 


11/72 


C 


22 


1,513.4 




00.0 


Georgia, Medical College of. 














Augusta 


12/72 


12/73 


13 


27B.2 




01 A 


Hahnemann Medical College of 














Philadelphia 


9/73 


C 


12 


665.7 




55.5 


Harvard Medical School-Francis A. 














Countway Library of Medicine, 














Boston, Mass. 


11/72 


C 


23 


1,599.5 


69.5 


Howard Univ., Washington, D.C. 


7/73 


C 


14 


131.B 


9.4 


Kentucky, Univ. of, Lexington 


11/72 


2/74 


16 


29.B 


1.9 


Loma Linda Univ., Loma Linda, Calif. 


9/73 


C 


6 


70.9 


11 R 


1 Ann«l«c Pniintu fPAlif ) Mpdipal 

I.O9 /AiHJ6l6b V^vUlliy \V^OIII./ lTlC\Jll<Ol 














Association 


12/72 


1/74 


14 


115.3 


B.2 


Louisville (Ky.), Univ of-Health Sciences 














Center 


4/73 


10/73 


4 


9.B 


2.5 


Maryland, Univ. of-School of Medicine, 














Baltimore, Md. 


2/73 


C 


19 


296.1 


15.6 


Massachusetts General Hospital, 














Boston, Mass. 


B/73 


C 


11 


40.4 


3.7 


Massachusetts General Hospital- 














Continuing Education 


10/73 


C 


11 


549.4 


50.0 


Massachusetts, Univ. of, Boston 


6/74 


C 


3 


13.2 


4.4 


Matthew Thornton Health Plan, 














Nashua/N.H. 


1/73 


6/74 


IB 


73.0 


4.1 


Mayo Foundation, Rochester, Minn. 


11/72 


B/73 


10 


142.5 


14.3 


McGill Univ., Montreal, Canada 


9/73 


12/73 


4 


104.5 


26.1 


Michigan State Univ., East Lansing 


11/72 


C 


22 


327.2 


14.9 
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Table 1 (Continued) 
Summary of Network Use 



UMr 


Usage Period 


TImt Used 


MMn Hours/ 
Month usage 


Start 


Continuing^ 
or Ending 
Date^ 


Months^ 


Hours 


... 
Michigan, Univ. of— School of Nursing, 












Ann Arbor 


3/73 


C 


18 


154.8 


o.o 


Michigan, Univ. ot— bcnooi ot 












Dentistry, Ann Arbor 


5/73 


8/73 


4 


23.1 


D.o 


National Institutes of Health, Clinical 












ucnter, Detnesoa, ivia. ^ 


1/73 


6/73 


3 


25.4 


O.D 


National Library of Medicine, 












Lister Hill Center , 


1 1/79 


c 


99 


1fi7 1 


7.6 


Naval Medical Research Institute, National 












Naval Medical Center, Bethesda, Md. 


1 I//Z 


r 


ZO 


IZD.Z 


5.4 


Nebraska, Univ. of. Medical Center, 












Omaha 


1 1 /TO 
1 I//Z 




zz 


ooe o 


10.3 


New England Deaconess Hospital, 












Boston 


11/72 


C 


22 


146.7 


6.7 


New York, State Univ. of, Buffalo 


4/73 


C 


15 


228.5 


15.23 


New York, State Univ. of-Children's 












Hospital, Buffalo 


8/74 




1 


2.2 


2.2 


New York, State Univ. of. 












Downstate Medical Center, Brooklyn 


4 4 /to 

1 1/72 




oo 

22 


rtOT f\ 

927.9 


42.2 


New York, State Univ. of. Stony Brook 


4 4 /TO 

11/72 




oo 

22 


< CO T 

152.7 


6.9 


New York, State Univ. of. Upstate 












Medical Center, Syracuse 


11/72 


C 


22 


407.7 


18.5 


Northwestern Univ., Chicago 


11/72 


1/74 


15 


573.3 


38.2 


Ohio, Medical College of, Athens 


1 1 /79 
1 1/ /Z 


19/79 
IZ/ /Z 


o 
Z 


1 


.7 


Ohio State Univ., Medical School, 












Columbus 


1 1 /79 
1 1 / /Z 


p 


1 7 
1 / 


OO.O 


2.2 


Oklahoma Univ., Norman 


5/74 


5/74 


1 


45.2 


45.2 


Oregon, Univ. of-School of Medicine and 












School of Nursing, Portland 


3/73 


C 


14 


1,117.4 


79.8 


Pacific, Univ. of— Pacific Medical Center, 












San Francisco 


2/74 


C 


7 . 


28.7 


4.1 


Pennsylvania, Medical College of. 












Philadelphia 


3/74 


C 


8 


44.6 


5.6 


Pennsylvania, Univ. of— School of 












Medicine, Philadelphia 


11/72 


C 


22 


2,583.7 


117.5 



■ (Continued) ■ 
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Table 1 (Continued) 
Summary of Network Use 



• 

User 


Usage Period 


Time Used 


Mean Hours/ 
Month Usage 


Start 


Continuingf 
or Ending 
Dateb 


Months^ 


Hours 


Pittsburgh, Univ. of-School of Medicine, 












Eye and Ear Hospital 


3/73 


C 


18 


517.4 


28.7 


RAND Corporation, Santa Monica, Calif. 


3/74 


5/74 


3 


8.3 


2.8 


nocnester, Univ, of, noc?iester, N.Y. 


11/72 


1/74 


15 


391.1 


26.1 


Rush Medical College, Chicago 


4/73 


8/73 


5 


83,7 


16.7 


Rutgers Univ., New Brunswick, N.J. 


5/74 


C 


4 


27.7 


6.9 


South Carolina, Medical Univ. of. 












Charleston, 


5/73 


C 


16 


654.6 


40.9 


Southern California Univ. of. 












Los Angeles 


11/72 


1/74 


15 


531.2 


35.4 


oouincrn Illinois liiiiv., uarDonuaie 

* 


5/73 


C 


16 


530.2 


33.1 


Stanford Univ.-School of Medicine, 




1 








Stanford, Calif. 


1 1 /79 
1 1 / /z 




zz 


1 ,Uoo.U 


46.9 


Texas College of Osteopathic Medicine, 












Fort Worth 


ft/?*? 
o/ /o 




1 o 


400.U 


35.7 


Texas Tech Univ., Lubbock 


10/73 


12/73 


3 


16.0 


5.3 


Texas, Univ. of, San Antonio 


11/72 


C 


22 


1,121.1 


51.0 


Texas, Univ. of, Galveston 


3/73 


1/74 


11 


317.4 


28,9 , 


Texas, Univ. of, Houston 


a/ /o 


1 /7A 


D 


1 CO A 


32.5 


Texas, Univ. of-Southwestern 












Medical School, Dallas, Tex. 


1 1 /79 


1 HA 


1 D 


017.4 


32.3 


Tufts Univ., Medford, Mass. 


R/7'? 
0/ /O 


p 


1 c 
ID 


170.8 


10.7 


Utah, Univ. of. Salt Lake City 


1 //o 




1 o 


389.1 


21.6 


Vanderbilt Univ., Nashville, 












Tenn. 


6/73 


8/73 


3 


17.1 


5.7 


Virginia Commonwealth Univ.- 












Medical College of Virginia, 












Richmond 


1/73 


7/74 


19 


1,913.2 


100.7 


Wa^indtnn Hn<nltal f^ntpr 
f f floi 1 1 1 ivi i nu^piiai ociiici. 












Washington, D.C. 


11/72 


C 


22 


335.5 


15.3 


Washington Univ., St. Louis, Mo. 


11/72 


C 


22 


961.6 


43.7 


Washington, Univ. of, Seattle 


11/72 


C 


22 


3,662.2 


166.4 


Wayne State Univ., Detroit, Mich. 


3/73 


4/73 


2 


64.9 


32.5 


Wisconsin, Medical College of. 












Milwaukee 


1/73 


C 


18 


473.6 


26.3 


Wisconsin, Univ. of, Madison 


6/74 


C 


3 


4.3 


1.4 



'Continuing participants are indicated by a "C." 
^ut-off date for data analyzed is August 31 , 1974. 

^Based on nunriber of months of actual usage rather than a straight calculation of start and end dates. If there were 
thrta or more consecutive months of nonusage, these were not counted. 
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could have been anticipated. Although the learning resource was much in demand, it 
failed to stimulate any broad curriculum reform. 

However, the CAI-Network utilization did benefit greatly from curricular refornjs*. 
and educational innovations that were already in place in many user institutions. Many of 
the user institutions had previously shifted from a high lecture orientation to a hi^ 
emphasis on self -directed study. These institutions had excellent facilities for self-directed 
learning, and were anxious to provide still another modality in addition to video 
cassettes, audio tapes, heart sound synthesizers, and other devices. 

Institutions with a traditional lecture-oriented, lock-step curriculum reported 
difficulty in scheduling CAI usage. The computer tutorials were in no case used as final 
examinations. Therefore, the utilization of CAI did not have any effect on established 
procedures for grading. 

The Network Experience has exposed administrators, faculty, staff, and students to 
the possibilities of the computer as a learning medium. When individual institutions made 
a decision to terminate Network involvement, students invariably were vociferous in their 
objections to losing this learning tool. 

Faced with the termination of the experimental Nework in May 1975, faculty 
members have been asked to review their thinking and to establish long-term policies 
toward this new media. However, only a few schools have made firm alternative plans for 
continuing CAI without the availability of the national Network. Many of them hesitate 
to make the heavy capital outlay for equipment and staff required to run their 
own facility. 

Some institutions are developing network substitutes. The University of Washington 
is working with the University of California at Davis on the use of a Meta compiler to 
translate their WISP Coursewriter programs to a language of the PDP-10 which is on 
order. George Washington University has acquired a minicomputer system and is gradually 
translating some programs in the CAI library to that system. 



IMPACT ON CURRICULA 

The CAI materials were used, with few exceptions, as a supplementary learning 
resource. While the CAI learning materials were perceived by both faculty and students to 
be a valuable educational addition, as measured by student usage and opinion surveys, 
they did not bring about any m^or changes in course objectives or content. The CAI 
materials served mainly as enrichment, or supplements, to their regular program. Third- 
year and fourth-year students had an opportunity to diagnose medical problems in coma, 
jaundice, abdominal pain, diabetes, heart irregularities, and other diseases. Computer 
simulations guarantee that students will "see" a large variety of cases with different 
symptoms and prognoses. 

At the University of Pennsylvania the discussion of the CAI clinical diagnostic 
exercises was a m^'or component of a course on the introduction to clinical medicine. 
These programs enabled the students to develop problem-solving skills in a way not 
possible in the traditional "making-the-rounds" with a faculty member. While the CAI 
programs do not supplant the rounds-making experience, they have demonstrated their 
ability to augment this method.^ 



A future direction mty be the development of a clinical core curriculum which is performance- 
based (i.e., the ttudent** ability to diagnose and treat medical problems is measured), and uses the com- 
puter as a major educational tool. 
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Most student usage of the CAI library was strictly voluntary. That is, on their own 
initiative they sought the use of the CAI library. (This practice supports the fact that 
students are highly goal-oriented and self -motivated by the time they reach medi- 
cal school.) 

Ohio State's tutorial learning units are most effectively used in conjunction with 
specific courses. Many of the CAI learning modules, when used in conjunction with 
textbooks, could substitute for lectures. However, few institutions revised curricula to 
make use of the OSU materials in this way. At the Medical College of Wisconsin, a series 
of Ohio State's anatomy modules was made an integral component of the gross anatomy 
course unit. 

In summary, the CAI library was used (a) as an augmentation of clinical experience, 
(b) as a self-assessment tool in basic science subjects, and (c) as a supplement to regular 
course work. 



IMPACT ON FACULTY 

Demonstrations were carried out in over a hundred institutions,, at which ^rme 
faculty had an opportunity to try out the learning materials. Many of the faculty 
members were so intrigued that they spent hours at the terminal going through the 
materials. While they were quick to point out deficiencies, they generally reported 
favorably on overall quality. 

Ohio State requested that its users evaluate the suitability of its learning modules 
prior to initiating use. The college counseled the user institution to establish faculty 
committees to review learning objectives of the learning units. Therefore, many faculty 
members were brought in on some aspect of the program even though they were not 
active creators of CAI materials. 

As a condition for receiving use of the Network facilities, user institutions were 
asked by the Lister Hill Center to prepare CAI materials to be added to the library. The 
medical school faculty was expected to develop these materials. It was hoped that they 
would become involved with tiie total process. However, the faculty at most institutions 
maintained a hands-off attitude toward CAI. They generally endorsed the idea of using 
the learning resource but left it up to the students as to how they would use it. 

In many of the schools that were the largest users (e.g., UCLA and Stanford), 
faculty members were not interested in becoming CAI developers, since research was their 
primary interest. 

Only in the final year of the experiment has an interest developed in authoring 
materials. Ohio State University has worked closely with the University of Pittsburgh Eye 
and Ear Hospital to develop learning units on diseases of the eye. CAI course content is 
being developed at the University of Pittsburgh, and the programming is taking place at 
Ohio State. In December 1974, Ohio State University reported that a total of 53 
instructional units were being developed by Lister Hill Network users. 

The Network Experiment did not provide funds for faculty release time to perfom 
development activities such as authoring course materials or devising new computer- 
oriented curricula. These activities are extremely time consuming and, in addition, require 
specialized computer skills. Therefore, the lack of faculty involvement was not surprising. 

Typical faculty attitudes regarding the Network Experiment and CAI materials are 
reflected in the following excerpts from copies of the Bimonthly User Report: 
"Demonstrations to faculty have, for the most part, been met with 
varying degrees of interest but as yet little commitment. However, 
there has been one encouraging development. A neurologist, tha 
chairman of the second year Neural Sciences Group, who has been a 
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leader here in innovative teaching, is prepared to utilize CAl heavily 
in his course. Since he plans the development of a modular sequence 
of course topics with student self-pacing, he would like to use a 
program such as UIMC's-CRIB to store a large number of test items 
which students could use both to monitor their own progress through 
material and approach as a criterion test at the completion of each 
course module/' 

Letter from Paul L. Grover, Jr., PhD,, Assistant Professor, 
Division of Medical Education and Communication^ 
University of Rochester, School of Medicine and Dentistry, 
March 6, 1973, 

"Most faculty members agree that time is the greatest factor in 
requiring use of CAL With the present curriculum load for first- and 
second-year students, it is difficult to find a block of time to take 
advantage of those programs considered valuable. There have been 
few who have found no merit in the system at all, and the verbal 
consensus of the faculty seems to be in favor of a teaching tool so 
well accepted by the students.*' 

Texas College of Osteopathic Medicine, Experimental Network 
Users Report, March 1, 1974, for the period January 1 - 
February 28, 1974. 

"Although thirty of these letters were sent out, unfortunately the 
faculty response has been poor and the materials have only been 
demonstrated to approximately ten faculty members.'' 

Letter from Robert F. Johnston, M.JD., Director, Division of 
Pulmonary Diseases, The Hahnemann Medical College and 
Hospital of Philadelphia, November 6, 1973, 

"Faculty usage has declined as most have had one or more 
demonstrations. Several instructors have returned to the terminal on 
their own and it is this group that I hope to interest in more 
substantive CAI input." 

Trial Users Report from Mirilyn Margon, Staff Development 
Specialist, Department of Educational Resources and Devel- 
opment, Southern Illinois University, School of Medicine, 
December 26, 1973. 

"Up to this time we have interviewed all faculty . . . users of 
the programs. 

"The Pharmacology faculty has indicated that, in their ^aew, several 
of the MGH programs provide an excellent means of bridging the 
disciplines of pharmacology and th^apeutics. 

"Introduction to the Patient (Physicah^Oiagnosis), the faculty has 
expressed some interest in considering UIMC-CASE as a means of 
inbroducing students to the experience of interviewing patients. 

"Faculty of the Department of Medicine concerned with the cHnical 
clerkships and third- and fourth-year electives are enthusiastic about 
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the capability of the programs to instruct the students in the 
approach to the management of disease processes (MGH programs)," 

College of Physiciam and Surgeons of Columbia Univenity, 
August 3, 1973, Report on use from Norman Kahn, JD,D,S,, 
PhD,, Pharmacology and J. Thomas Bigger, Jr., Associate 
Professor of Medicine and Pharmacology, 



IMPACT ON STUDENTS 

Network materials received the most usage from students, usually on their own 
initiative. It was evident from this heavy continuing use that students perceived the 
learning experience to be important. 

Some user institutions attempted to keep data on niunber and types of students 
using the resource, but had difficulty getting users to complete survey forms. Data on 
system usage, however, are available from the contractors, Massachusetts General Hospital 
was the only Network node which gathered usage data by type of student The 
Massachusetts General data cannot indicate how many students used its system, since 
individual students did not have unique identifiers. Its data show how many times a 
particular category of student used the programs. 

Table 2 shows the usage time of the Massachusetts General Hospital system, by class 
of user at UCLA and Stanford, two major user institutions. 



Table 2 



Usage Time of Massachusetts General Hospital System, 
by Class of User (11/1/7210/1/74) 



Institution/Class of User 



Ustgc Timt* 



UCLA 

Med 1 (pre-clerkship) 

Med 2 (clerkship) 

House Physician 

Nonmedical 

Auxiliary Medical 

Physician 

Nurse 

Fellow 



747 
631 
299 
128 
79 
55 
22 
11 



Stanford 

Med 2 (clerkship) 
Med 1 (preclerkship) 
Nonmedical 
House Physician 
Physician 
Auxiliary Medical 
Fellow 
Nurse 



254 
235 
45 
29 
22 
14 
6 
0 



'Rounded to the nearest hour. 
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No provision was made in the Network Experiment for a systematic assessment of 
the impact of the materials on student learning. Each user institution was expected to 
provide its own evaluation of the materials and their use. However, only a few 
institutions were able to complete any kind of systematic study. Furthermore, each 
institution that performed some kind of study performed it in its own way. Objective 
summary data regarding student learning are available for only a few institu- 
tional evaluations. 

Surveys of faculty and student opinion were conducted by five institutions: Medical 
College of Virginia, Mayo Foundation, Albany Medical College, State University of New 
York Downstate Medical College, and the University of Pittsburgh. Findings were sub- 
mitted to the Lister Hill Center. 

Students at Albany Medical College were specifically asked to rate the educational 
value of specified CAI materials. Summary results of this study are shown in Table 3. 

Table 3 



Ratings by Albany Medical College Students on 
Educational Value of Specified CAI Materials (October 1973) 



Program Number and Namt 


Number of 
Times Used 


Average Bating 
(Scale, M0)» 


1. 


Abdominal Pain 


3 


7.67 


2. 


Anticoagulant Simulation 


5 


8.40 


3. 


Cardiac Simulator 


1 


4.00 


4. 


Cardiopulmonary Resuscitation 


15 


8.67 


5. 


Coma 


4 


9.00 


6. 


Diabetic Ketoacidosis 


8 


8.63 


7. 


Digoxin Dosage Advisc/T 


1 


6.00 


8. 


Idiopathic Respiratory Distress in the Newborn 


2 


8.00 


9. 


Jaundice 


"3 


8.33 


10. 


Pediatric Cough and Fever 


1 


9.00 


11. 


A Bird's-Eye View of Pediatrics 


6 


. 7.50 


12. 


Acid Base Balance 


5 


8.70 


13. 


Anticoagulant Medication 


2 


8.50 


14. 


CAI Demonstration Games 


6 


2,00 


18. 


Introduction to CAI 


1 


3.00 


20. 


Juvenile Diabetes 


1 


8.00 


30. 


Reading the Medical Record 


1 


2.00 


37. 


CASE 


12 


9.08 


38. 


CRIB 


8 


8.00 



'Highest possible score. 10, 



Controlled experiments to determine the effects on student performance were too 
costly in faculty time to be carried out at most institutions. One study was conducted 
with 34 resident nurses at Cape Cod Hospital (Hoffer etaL, 1974). In this study, 
22 nurses using 90 hours of computer time made statistically signiiBcant improvements in 
a test of tlieir knowledge of cardiopulmonary resuscitation. A control group, of 12 nurses 
who had no access to the computer programs, showed no improvement in knowledge. 



16 



2G 



IMPACT ON LIBRARIES AND MEDIA CENTERS 



The Li«ter Hill Network terminals were located in a variety of places (emergency 
rooms, hospital corridors, hospital wards, libraries, learning resource centers, medical 
association headquarters, student and faculty lounges, laboratories, etc.) 

When terminals were moved from inconvenient places into a conspicuous location in 
the library, their usage greatly increased. A list of terminal locations by type and number 
is provided below: 

Biomedical Libraries~25 

Student Biomedical Labs (planned)— 17 

Learning Resources Centers (multi-disciplinary lab)— 16 

Computer Laboratories and Centers— 12 

Medical School Departments (Surgery, Pathology, Ear, Nose and Throat)— 16 

Terminal Rooms— 5 

Conference Rooms-4 

Offices of Medical Education--4 

Student Study Areas- 3 

Emergency Rooms— 3 

Offices of Continuing Education— 3 

Resident Lounges'~2 

Ward Rooms— 2 

Cardiac Care Units— 2 

Hospital Library— 1 

Family Practice Clinic— 1 

Health Testing Unit— 1 

Unspecified— 29 

Individual librarians interviewed (e.g., Gloria Werner, Assistant Biomedical Librarian, 
UCLA) were very enthusiastic about the potential of CAI service. The UCLA library 
constructed special facilities for CAI library use— sound-proofed enclosed study areas 
where groups of students could work at the terminals without disturbing other 
library activities. 

Ms. Louise Darling, the Biomedical Librarian at UCLA, recognizes CAI as an 
important service of the library but also feels that it is not the library's role to become 
involved with curriculum matters that inevitably arise whenever CAI is used to augment 
traditional instruction. Because of this problem, UCLA established a special CAI Faculty 
Curriculum Committee to review UCLA's long-term commitment to CAI. 

When librarians and media center directors were questioned about their interest m 
having their own local system (desktop computer or intelligent terminal) they liked the 
concept but were somewhat apprehensive about possible technical problems and the 
availability of instructional programs. The idea of a CAI program cassette library was 
particularly appealing to the librarians since they were already using cassettes for video 
and tape instruction. 
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Chapter 3 

INSTITUTIONAL VARIABLES AFFECTING UTILIZATION OF NETWORK 



The CAI Network Experience has pointed up the following key variables that 
determine the quantity and quality of usage of CAI materials in an institution. 

(1) Curriculum Integration . This factor is of paramount importance for long-term 
commitment. There were only scattered instances of any attempt to integrate the MGH 
clinical encounter programs into the clerkship curriculum. Even at UCLA, the largest user 
school, thare was virtually no faculty interest in becoming involved with CAI. Without 
thii faculty involvement, the schools will not back large budgetary expenditures for CAI. 

The Ohio State materials, because of their tutorial nature, require some form of 
curriculum integration. This may explain why Ohio State's program usage has been far 
less than MGH's program usage. 

Several user institutions stated that the principal reason for terminating their 
participation in the CAI network was that they could not justify the network charges if 
the materials were to be only an appendage to the regular curriculum. 

(2) Faculty Type. Schools that had a full-time faculty who were devoted to 
teaching and research (and who did not have an outside private practice) were more 
amenable to commiting time to exploring CAL 

At the University of Washington, which has a full-time faculty, there are as 
many as 25 faculty members working on and developing CAI modules, including the 
extensive reworking of the visual materials that accompany CAI programs. This involve- 
ment contrasts with UCLA faculty members who have heavy involvements in private 
practice and thus participated very little in the use of the Lister Hill Network. The 
faculty committee at UCLA voted against continuing the project after termination of 
Lister Hill support. 

(3) Educational Philosophy and Program . This variable greatly influences the usage 
level. The CAI learning materials, particularly those from MGH, are ideally suited for use 
in an unstructured curriculum. Typically, high levels of usage would be found where 
lecture attendance is not required and the students are expected to study on their own. 
Stanford University is the most striking example where the faculty considered it highly 
desirable that the students have several different learning modalities available to them. 
Even though Stanford does not have a formal Independent Study Program (ISP), as do 
Ohio State and the University of Washington, the ISP modules are extensively used for 
self-assessment by pre-clerkship students weak in basic sciences. 

In schools that have a highly structured learning approach, faculty members often 
stated that schedules were so tight there simply was not enough time for the students to 
use additional CAI instructional units. 

CASE EXAMPLES 

The interaction ofv these institutional variables can be seen in these case examples, 
which are discussed in detail on the following pages. 
UCLA School of Medicine 
University of Washington School of Medicine 
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Stanford University School of Medicine 
University of Southern California School of Medicine 
Los Angeles County Medical Association 
University of Oregon 

University of California at San Diego, School of Medicine 
Medical College of Virginia 



f 29 



INSTITUTION VISITED: UCLA, School of Medicine 



DATE OF VISIT: July 31, 1974 

INDIVIDUALS INTERVIEWED: 

Mrs. Gloria Werner 
Assistant Biomedical Librarian 
for Public Services 

Ms. Louise Darling 
Biomedical Librarian 

Dr. Bernice Wenzel 
Assistant Dean 
School of Medicine 
Office of Curricular Affairs 



Dr. Robert Robinson III 
Assistant Director 
UCLA Hospital and Clinic 

Dr. Alan Forsythe 
Health Sciences Computing 
Facilities 



The UCLA School of Medicine, as the largest single user in the Lister Hill network, might be 
expected to exhibit a cohesive sense of direction. However, attitudes on CAI value vary from high levels 
of enthusiasm to skepticism. 

A special advisory committee has recently been formed to establish a long-term policy regarding 
the future of CAI at the UCLA School of Medicine. Dr. Wenzel is chairing the committee for this study. 

Departmental chairmen at the School of Medicine are conservative; they want definitive proof of 
the educational value of CA!. The difficulty of establishing this proof is well known in CA^ pircles . 
Presuming that experiments were carried out using control groups, there is always the question as to 
whether some other method of instruction would have produced as satisfactory an effect. Although the 
prevailing attitude is noncommittal, there has been some support, especially by Dr. Lampson, Director of 
the UCLA hospital. The library has received the requisite money from extramural sources to meet 
current network charges. 

Until now student use of CAI has been entirely library oriented. The CAI programs have not been 
integrated into the curriculum. Yet there is evidence that the students have found it more than just an 
interesting diversion since they have continued to make use of it over an extended time. Students have 
indicated to the faculty that UCLA is behind the times for not having CAI as a regular part of 
the curriculum . 

Although the CAI programs have been a popular library offering, Ms. Darling feels that they 
should not be the sole responsibility of the library. In her view, the faculty should involve themselves in 
curriculum-related aspects of the CAI program. It was suggested that the CAI network might become a 
permanent National Library of Medicine service similar to MEDLINE. At the same time, it was felt that 
the network should not be continued indefinitely in an experimental state. 

A major problem is that the faculty does not have the time to devote to CAI. Just now CAI is a 
stepchild insofar as any real long-term commitment is concerned. 

The advisory committee has many basic issues to work out about the possible future role of CAI. 
Since the overall commitment to CAI is in question, little thought is being given to the immediate 
problem of network termination. However, a negative feeling about the feasibility of developing a 
satisfactory alternative to the national network persists. 

Dr, Robinson expressed grave concern about the problem of updating and maintaining a CAI 
library. He feels that the initia', language conversion is a major undertaking itself; additionally there is 
the problem of ever-changing CAI program content. Therefore, he feels that the best solution is through 
an interactive library where the user could dial-in for the latest version of the CAI program which would 
then be run on a local computer. The type of facilities that Dr. Robinson envisions are similar to those 
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found on the experimental ARPA network where programs or files created at one site are shifted to 
another site for execution. 

During the interviews, the need for an evaluative system for CAI programs was continually 
mentioned. If such a system was in effect it appears that greater faculty support could be obtained. At 
the same time, UCLA has no money for evaluation and would require outside support to conduct 
such evaluation. 

UCLA has several options should the network be discontinued: 

(1) Purchase intelligent terminals. 

(2) Purchase mini-computers. 

(3) Utilize large campus computers.^ 

(4) Join a re^onal network. 

(5) Individual negotiation with vendors. 

(6) Drop CAI. 

In the on-site interviews, the concept of a regional or state CAI network appeared to be most 
acceptable. The UCLA School of Medicine already serves as a regional biomedical library and it would 
be natural to extend this-to CAL 



*An IBM 360/91 computer is currently operating in the Health Sciences Canter. The current ch(ir,;5s ore 
approximately $3.50 per connect hour. The main problem with using this facility for local or regional use is thr.t it is 
principally used as a research computer. This nneans that two or three times a day the computer may bo down for 
software dtvelopment work. 

o 31 21 

ERIC • 



INSTITUTION VISITED: University of Washington, School of Medicine 



DATES OF VISIT: August 8-9, 1974 

INDIVIDUALS INTERVIEWED: 

Ralph E. Cutler, M.D. 
Director 

Washington Independent Study 
Program 

Gary E. Striker, M.D. 
Assistant Dean for Curriculum 



George Brengelmann, M.D. 
Faculty Member of CAI Committee 

Starkey Davis, M.D. 

Faculty Member of CAI Committee 



In contrast to other user institutions visited, the University of Washington's School of Medicine is 
a forerunner in integrating CAI into the mainline curriculum. The Washington Independent Study 
Program (WISP) is designed to permit the student to progress at his own rate through the pre-clerkship 
portion of the medical curriculum. This self-pacing permits a student to spend additional time on 
particularly difficult material or concepts while moving quickly through material he can command easily. 
The program is organized into instructional unfts (modules); objectives to be learned in each module; 
selected references to learning resources (books, journal articles, films, slides, tapes, etc.) which may be 
pursued to meet the objectives; a computer -assisted, self -evaluation exercise designed to help determine 
how well the objectives have been met; and the names of the faculty members who are responsible for 
consultation regarding the module. 

Since the majority of the students' activities in the WISP are designed to be pursued 
independently, there are few scheduled activities. Most days students are free to manage their own time 
as they choose; however, one or two days a week, there are scheduled activities (average 3-6 hours per 
week). Throughout the two-year program, students are free to take electives from the tradi- 
tional program. 

A student may seek consultation with a faculty member at any time. When students are satisfied 
that they have learned the material which was specified in the objectives for the module, they go to a 
computer terminal to take a computer-assisted, self-evaluation exercise which consists of questions and 
case studies. The computer reacts to the student's response with reinforcement, corrective hints, or 
tutorial coaching depending on the nature of the student's answer. A self-evaluation exercise is provided 
with each module and is based on the objectives for the module. It provides the student with regular 
feedback related to his attainment of the objectives. Upon satisfactory completion of the self-evaluation 
exercise, the student proceeds to the next module. 

Faculty evaluation of the student's performance occurs in the form of nine written objective 
examinations which are administered after a sequence of three to four modules. 

WISP students are volunteers who make a commitment to the program. The program provides an 
alternative to the lecture method for students who prefer to direct themselves. 

WISP was conceived about the same time as the Lister Hill Biomedical Data Communications 
network. The University of Washington's School of Medicine sent a special committee to Ohio State 
University to review and evaluate that medical school's Individual Study Program (ISP). The committee 
recommended that the Ohio State ISP modules be adapted for use in their medical school. Approxi- 
mately 20-30 faculty members participated in the decision and subsequent revision of the Ohio State 
materials. Revisions have varied from minor changes in sequence to complete rewriting of 
course modules. 

Faculty involvement in CAI at the University of Washington's School of Medicine is probably 
greater than any other school in the network. The primary reasons are: 
(1) Curriculum development has a high priority. 



•32 



(2) Grant monies are available for curriculum development through the WISP and 
Washington, Alaska, Montana, Idaho (WAMI) projects. 

(3) Faculty are not allowed to have a private medical practice. 

(4) There is an assistant dean for curriculum. 

Dr. Striker, Assistant Dean for Curriculum, feels that the computer must be integrated into the 
regular course of study if it is to remain an effective learning tool. This appears to be substantiated by 
the experiences of other network users like UCLA, where the continuation of CAI is up in the air. 

" The School of Medicine's plan for the next three to five years is to develop a curriculum in which 
clinical simulations will become an integral component of the clerkship years. The school's bias is 
towards quantitative assessment of student performance, and simulations offer this possibility for 3rd 
and 4th year students. 

The University is planning to use the WISP evaluation programs in their WAMI project which 
involves the rural communities in Washington, Alaska, Montana and Idaho. These states, except for 
Washington, do not have medical schools. The WAMI program is designed to provide medical education 
in remote locations. Course committee chairmen at the University of Washington meet with cor- 
rtsponding faculty at the WAMI sites to establish a common course content. Further, WAMI faculty 
members have participated in the revision of Ohio State's independent study programs. The WAMI 
program will be using a satellite system capable of two-way interactive conversation. 

Some independent study students have expressed anxiety over the experimental program. One 
student said, "I wonder if I am learning anything," The knowledge data base to which WISP students 
are exposed is actually far greater than that of their counterparts in a traditional program. Even though 
they are exposed to this wealth of information, they are nervous about acquiring the specific knowledge 
necessary to pass the final exams and the Boards. 

A major outgrowth of the independent study program is that faculty members have had to . 
integrate the learning resource materials (films, video cassettes, slides, audio tapes, CAI programs) into 
their thinking, and curriculum planning. . 

Not everyone on the faculty is an advocate of CAI. Dr. Brengelmann was vociferous in his 
objection to CAI. He maintains that, at best, some of the modules are routine, and, in addition, CAI is 
not particularly cost-effective. He further said, "The reading speed of an intelligent medical student is 
faster than a 30-character/second terminal, and people should work with people." 

Dr. Davis is more favorably disposed toward the computer, although he is not in favor of using it 
for something that can be done just as well with paper and pencil. Dr. Davis developed a series of 
computer clinical exercises to be used as part of the microbiology course unit. These exercises help 
students perceive the clinical relevance of basic scientific knowledge. At the end of each unit of study. 
Dr. Davis spends 2 to 3 hours discussing every question on the final exam with each student. He feels 
that individual conferencing is a more productive learning process than the time he used to spend 
lecturing the students. 
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INSTITUTION VISITED: Stanford University, School of Medicine 



DATE OF VISIT: August 16, 1974 

INDIVIDUALS INTERVIEWED: 

James B. Williams, Director James Steward, M.D. 

Division of Instructional Media Associate Dean for Student Affairs 



Stanford University has a highly unique environment. First, the university has been a forerunner in 
CAI research and development through the work of Dr. Patrick Suppes of the Institute for Mathematical 
Studies in the Social Sciences. 

Second, the School of Medicine is overwhelmingly a researchoriented institution, even to the 
point that students are expected to independently pursue their individual learning interests. Third, under 
the Stanford curriculum plan, the medical student is recognized as a graduate student inasmuch as he is 
allowed wide freedom to plan his own program. Dependence on didactic teaching is minimized. Instead, 
seminar methods and the' use of small groups, fostered by the physical arrangement in the multi- 
discipline laboratories and by the provision of many small conference and demonstration rooms, are 
emphasized. The student is actively encouraged to undertake independent study and research arid to 
widen his horizons, both medical and non-medical, through the judicious choice of study in other 
divisions of the university. This does not, however, imply a de-emphasis of the primary importance of 
the medical sciences. 

In the current curriculum, there are no required courses per se, classroom attendance is not 
mandatory, and listings of prerequisite courses are for guidance only . 

The Committee on Courses and Curriculum strongly recommends that the students be given an 
opportunity to try out the new modality, namely, the Lister Hill CAI network. This response is in 
complete agreement with their general philosophy of presenting the student with as many alternative 
methods of learning as possible. 

While the faculty is very much in favor of CAI use, they are not interested in diverting themselves 
from their research efforts to the development of their own library of CAI materials. For this reason 
Stanford would find a network library of CAI materials of special benefit. 

Even though the curriculum is completely unstructured, the medical school faculty is very much 
aware of the need to verify that the students have acquired fundamental knowledge of the medical 
sciences. Stringent final exams are given in each course unit such as pathology and anatomy, with "pass'* 
levels set even higher than the National Boards. 

Dr. Steward mentioned that consideration is being given to combining the present three National 
Board exams into a single exam to be given prior to the beginning of internship. This would introduce 
the problem that students would be called upon to remember basic science materials studied during the 
first two years of medical school. The utilization of CAI as a modality for review would be of decided 
advantage to the Stanford curriculum plan. 

Although the Stanford University School of Medicine attracts extremely intelligent students who 
are highly motivated, there is still a need for remedial education in the basic sciences. The School of 
Medicine has a significant percentage of minority students who are deficient in background knowledge in 
the basic sciences. These students were heavy users during the experimental period and appear to have 
found the Ohio State materials especially helpful for review and catch-up. 

A little over a year ago a Division of Instructional Media was established to consolidate 
technology-based educational use. The installation of the CAI terminal is an integral part of the overall 
Fleischmann Teaching and Learning Center. The CAI terminal is located in a lounge area just as you 
enter the center. What is striking is the informal atmosphere surrounding the terminal. The terminal sits 
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on a low table in front of a large couch. It was especially purchased with a video output jack so that 
the image can be displayed on a large TV monitor sitting in the corner of the lounge area. 

Dr. Steward and Mr. Williams both indicated that they feel that a national network is the best 
possible arrangement for the sharing of CAI materials. Regional networks, however, would be a good 
alternative. At the present time, "CAI users have no common project, just common problems." There 
has been little or no contact between network users. 



INSTITUTION VISITED: University of Southern California, School of Medicine 



DATE OF VISIT: August 2, 1974 

INDIVIDUALS INTERVIEWED: 

Nelson Gilman, Librarian Elaine Adams, PhD. 

Norris Medical Library Learning Resource Coordinator 

Roger W. Jelliffe, M.D. 
Associate Professor of Medicine 



The CAI programs were a phenomenal success at USC's School of Medicine, although subsequently 
the "patient" died. One terminal was installed in the library and programs from MGH, University of 
Illinois, and Ohio State University were made available. Because of the demand and the availability of 
only one terminal at the library, it was decided that scheduling appointments for terminal use would be 
the most efficient method. The terminal was located in the lower level of the library, in the library's 
group study area. In addition to the terminal at the library, there were several terminals at the hospital 
which were used primarily for other purposes, but were also made available to network users. 

Nearly $1,000 was spent on the initial publicity for CAI which had a large payoff in terms of 
subsequent student usage. A great deal of effort was expended in attempting to demonstrate the CAI 
programs to the faculty, but the faculty was negatively disposed toward CAI because they felt: 

(1) It was a dehumanizing educational experience. 

(2) Many factual errors were contained within the CAI programs. 

(3) Conflicts existed with the way that courses were taught at USC (sequencing of topics, 
emphasis placed on particular topics, etc.). 

On the other hand, many of the faculty members who took the time to come to the demonstrations 
liked the programs and found them stimulating . 

When Lister Hill announced the network charges, the Medical Education Committee reconsidered 
their CAI posture. They decided that the programs should undergo formal evaluation. In the interim, 
they withdrew from the network since the committee has given no further attention to the matter, CAI, 
in effect, was given a pocket veto. 

Dr. Jelliffe is the most knowledgeable staff member at USC in interactive computer programming. 
His interest in the CAI network was principally curiosity about the work at MGH. In his opinion, the 
program on digoxin dosage presents an oversimplified view of variables which affect dosage level. He has 
developed a sophisticated set of his own drug kinetics patient care programs. His aim is to have the 
computer determine drug dosages (e.g., the amount and rate for intravenous feeding). 

Dr. Jelliffe was, in general, critical of the CAI programs. (His comments on the MGH programs 
were at variance with almost every other doctor's assessment as to their value.) His focus is on patient 
care programs and not on teaching programs. He currently utilizes the G.E. Mark III communications 
network. This network is available by local telephony from nearly 350 cities in North America, Western 
Europe, Australia and Japan. Dr. Jelliffe stated that he found the G.E. network to be more reliable than 
the Tymshare network. During the period in which he used the Tymshare network^ the conversations 
were plagued by unexplained interruptions. 
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INSTITUTION VISITED: Los Angeles County, Medical Association 



DATE OF VISIT: August 5, 1974 

INDIVIDUALS INTERVIEWED: 

Ben Z, Krentzman, M,D, Milton G, Sheinbein, M,D. 

LACMA Computer Committee 
Chairman 



The Los Angeles County Medical Association (LACMA) visit provided an insight into some of the 
problems of utilizing technology for the continuing education of medical practitioners. Dr, Krentzman, 
when he initially considered using the network, envisioned the possibility of providing thousands of 
doctors (LACMA has approximately 10,000 members) with a 'lirst hand experience using a computer," 

Because of the computer's increasing role in all aspects of medicine, Dr. Krentzman feels that 
every physician shoul d have some basic computer knowledge . LACMA frequently holds large dinner 
meetings wf^ich are attended by 200-300 physicians. These meetings are held at the LACMA auditorium 
and at individual hospitals throughout the metropolitan community. Dr. Krentzman had planned to 
demonstrate the materials before such audiences using large screen projection of the terminal printout. 
He investigated the cost of portable projection systems and found one that costs approximately $5,000, 
The LACMA computer committee was approached about funding the purchase of this equipment, but 
voted it down. Dr, KrenUman feels that the committee's negative vote was a reflection of the senior age 
profile of its membership, 

Dr, Sheinbein. the chairman of the committee, provided a different perspective on what happened. 
When the committee first voted to use the network it was to be at no cost to LACMA, The availability 
of a rented terminal at the LACMA headquarters for CAI demonstration was given wide publicity. 
Dr. Sheinbein said that the request for funds came in the midst of an austerity mood at LACMA and it 
was felt by the committee that they could not afford to invest in the equipment at that time. 
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INSTITUTION VISITED: University of Oregon, Medical and Dental Schools, 

School of Nursing and Health Sciences Center 



DATE OF VISIT: August 6, 1974 

INDIVIDUALS INTERVIEWED: 

Health Sciences Center 

Gerald W» Miller, Professor 
Educational Research & 'Medical 
Psychology (School of Nursing) 

Evelyn Schindler, Coordinator 
Masters Program 

James McDonald 
Laboratory Associate 

School of Nursing 

Marcella Cate, Associate Professor 
Medical-Surgical Nursing 

Dorothy Elhart, Chairperson 
Medical-Surgical Nursing 

Bernice Jones, Chairperson 
Pediatric Nursing 

Loretta Myers, Chairperson 
Maternal-Child Nursing 

Doris Schindler, Assistant 
Professor, Psychiatric Nursing 



Dental School 

Patrick J, Reynolds, PhD. 
Assistant Professor of 
Physiology 

Carol Jenkins 

Assistant Professor & Librarian 
Library 



School of Medicine 

M. Roberts Grover, Jr., M.D. 
Associate Dean 

John T. VanBruggen, M.D. 
Director of Teaching Laboratories 
Professor of Biochemistry 

Robert E. Reichart, Ed.D. ^ 
Director 

Office of Medical Education 

Walter R. Petersen, Director 
Computer Center 

Heather G. Rosenwinkel 
Assistant Professor, Library 



Dr. Miller of the School of Nursing arranged for visits with a wide spectrum of people involved 
with CAI at the University of Oregon. Individuals interviewed included administrators, coordinators, 
faculty and students. 

The School of Nursing has made more progress with CAI technology than has the School of 
Medicine, even though the CAI programs available through the network were not specifically developed 
for nursing students. 

It is interesting to note that the Dean of the School of Medicine has provided the School of 
Nursing with financial support in CAI even though the School of Medicine itself has made little progress 
in technology*based education. 

School of Nursing students used both the MGH clinical assimulations and a number of programs 
from Ohio State. A Bird's Eye View of Pediatrics was especially popular. 

Students complained that some of the programs were not appropriate to their curriculum (e.g., 
nurses are not supposed to make medical diagnoses). However, in one instance a student nurse was able 
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to mikt » diagnosis from what she learned from the CAI program, Major Tranquilizers in the Pre-Med 
Program. She was able to apply this knowledge to a psychiatric patient who appeared to be catatonic. 
She thought the symptoms were drug-induced. When she suggested this to the doctor, he discontinued 
the drug. The next day, the patient was radically improved. 

The School of Nursing has just obtained a Wang programmable calculator under a Title VI Grant. 
The Wang is a desktop computer system which can be time shared among several terminals. The current 
configuration cost is $13,500. , 

The options open to the School of Nursing at the University of Oregon when the Lister Hill 
Network terminates are: 

(1) Upgrade the Wang system with disk drives, additional memory, ports, and a card reader. 

(The estimated additional cost would be $36,000.) 
12) Use the campus computer, an NCR Century 251. This computer uses BASIC language, 
but the School of Nursing would probably not attempt a conversion effort from MUMPS 
and Coursewriter without outside support. 
(3) Use a 360/50, currently in the Chancellor's Office, which is soon to be replaced and 
couW be used as a CAI computer. External support, however, would be required to 
make the system fully operational, including the purchase of the Coursewriter compiler. 
14) Use commercial or public timesharing facilities. (A serious drawback would be the high 
cost per terminal hour.) 

The meeting with School of Medicine faculty and administrators was surprising because of their 
generally negative attitude when contrasted to the level of interest found in the School of Nursing. The 
School of Medicine had no plans for the future for CAI, and their current network experience did not 
appear particularly rewarding. 
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INSTITUTION VISITED: University of California at San Diego 

School of Medicine 

DATE OF VISIT: August 20, 1947 

INDIVIDUAL INTERVIEWED: 

John Sewell, Supervisor 
Learning Resource Center 



The philosophy of Dr. Roger Marchand, Coordinator for Learning Resources, is one of not 
imposing technological educational aids on faculty. It is a laissez-faire policy of making these innovations 
available for individual faculty evaluation. Thus far, only a few members have availed themselves of 
thest materials; most of the use has been by second year medical students using Ohio State's ISP 
materials. Students in their clinical years are located in a hospital about 25 miles from the campus. 
However, there are plans to install a terminal at the hospital which will then be available for MGH 
clinical simulations. 

The School of Medicine has no formal plans for continuing access to Ohio State materials. 
However, Mr. Sewell has indicated that the most practical arrangement would be to tie into some other 
California user institution. He heard that the University of California at Irvine has received the Ohio 
State programs and is in the process of putting them up on their computer. He is also interested in the 
MUMPS interpreter being developed at the University of California Davis School of Medicine on the 
Burroughs 6700 since all the University of California schools, except one, have the Bur- 
roughs 6700 computer. 

A particularly interesting aspect of the visit was the opportunity to become familiar with the 
Datapoint intelligent terminal that is housed in the learning center. This terminal is currently being used 
for test scoring and analysis. Students enter their answers on a special Inswer card which is then 
optically scanned and the results are scored on a tape cassette built into the terminal. The answers can 
be displayed on the terminal device but it does not have sufficient computer power to perform the 
analysis. On command the data is sent to a larger campus computer for processing. Basic programs are 
also written at the terminal for execution on the larger computer. 

This school characterizes a typical user institution: They neither aggressively advocate CAI, nor are 
they passive toward its role in the educational process. The learning lab has an extremely small budget 
(around $5,000 annually) for the purchase of supplies and equipment. As a result, the learning resource 
lab facilities are sparse in contrast to laboratories found at other user sites like Stanford and UCLA. 
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INSTITUTION VISITED: Medical College of Virginia 



DATE OF VISIT: 



September 30, 1974 



INDIVIDUALS INTERVIEWED: 



David Hopp, Ph.D. 
Educational Planning and 
Development Program 



John Vorvell, M.D. 
School of Basic Sciences 
Department of Anatomy 



John Wergin, Ph.D. 
Educational Planning and 
Development Program 



Nelson Young, M.D. 
School of Medicine 
Department of Pathology 



The Medical College of Virginia's termination of its network affiliation seemed to be enigmatic, at 
first glance. After a period of intense activity (MCV was the fourth largest user in the network during 
1973) a decision was made to end the service. The peak month of usage was 280 contact hours. 

During the interview it became clear that cost was the single most important factor influencing the 
decision to terminate, although a closer analysis revealed other underlying factors. A private line 
between Richmond and the nearest communication computer (TYMSAT) in Washington, D.O., cost the 
school $400/month, and MCV dropped out when Lister Hill instituted connect-time charges. Adminis- 
trators at the institution expected the charges to soon climb to $10 per connect hour which would have 
resulted in a cost of $2,000/month. The question arose as to who would pay the bill. Dr. Young stated 
that $1,000 hard cash was a lot of money! 

Some of the specific reasons why this expenditure, in the eyes of the staff, is not justified are: 



(1) Coverage will likely continue to be small. According to the Media Center librarian, these 
programs are usee primarily by a small, core group of students and interns. 

(2) Without a specific reason for existence, the CAI programs will continue to have trouble 
attracting the interest of faculty members and students. This is especially true once their 
novelty has worn off. 

(3) The benefits of these programs will continue to be difficult to measure and translate 
into learning outcomes, and hence the funds required for their maintenance will be 
difficult to justify. 



Another factor in the forefront of the Educational Planning and Development Program staffs 
thinking is that the CAI packages have not been incorporated into the MCV curriculum. The faculty, 
with a few exceptions, has not been inclined to make a real commitment to CAI utilization. (In the 
nursing school, the program SHOCK was assigned, but the medical school faculty did not assign program 
use as a classroom lab.) A few faculty members were initially interested in the programs but did not 
follow through. One instructor even went so far as to commit himself to participate in a controlled 
evaluation, but he never quite found the time for the experiment. He had planned to revise his own 
self-instructional package to fit with one of the MGH programs so that the two self-study methods could 
be compared. 

MCV conducted extensive questionnaire studies to ascertain the overall effectiveness of CAI. The 
average user response, by supplier institution, is contained in Table 4. 

It is clear from the data that the students have a strong preference for this learning modality 
although they were Jess sure of their feelings as to how much knowledge they gained from it. When the 
students returned to campus for the new academic year, they were surprised to find that the CAI 
terminal had been removed. They were so upset that they petitioned the administration to reinstate it. 
Some of their comments were: 



/f W9S very, very useful to help develop clinical /udgments. 
There was much valuable information in that computer. 
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Table 4 



Average User Response, by 
Supplier Institution and Item Type' 





Supplier Institution 




MGH 


UIMC 


OSU 


Item Type 


(N = 17)^ 


(N = 14) 


(N « 23) 


General Attitude 


4.11 


3.72 


3.86 




(N'= 9) 


(N'=11) 


(N'= 9) 


Knowledge Gained 


3.80 


3.40 


3.80 




'(N'=14) 


(N'=15) 


(N'=14) 


Mechanics 


3.69 


3.67 


3.58 




(N'=16) 


(N'=17) 


(N'=15) 



^Items scaled between 1 and 5, least to most favorable: 3.0 indicates neutrality. 
This means that for positively-worded items, "strong agreement" was given a score of 5; 
for negatively-worded items, "strong disagreement" was given a score of 5. 

'^N - number of users; N '» number of items. 



Source: Worgin, 1973. 



We v^re ffo/ng to be married. 

Taking the computer out was a baclcward step. 

it combined enjoyment with iearning. 

Aithough i reaiize that the computer is expensive, i feei that its use by students is extremeiy 
beneficiai. 

Some of these programs provided very good reviem for Part / of the Nationai Boards, 

A summary of the student judgments of the utility of the CAI programs is contained in Table 5. A 
chi-square analysis of these data indicates a significant tendency for different supplier offerings to have 
different learning benefits, at least as perceived by the respondents. The MGH programs tended to help 
develop problem-solving skills, while the OSU programs tended simply to transmit information. 
Responses to the UIMC programs are mixed between the two extremes; this latter finding is probably 
due to the difference in orientation between the CRIB programs (facts) and the CASE programs 
(problem solving). 

Additional information on the type of user of MGH programs is available. It is notable that the 
MGH programs were found to be beneficial by a wide spectrum of users, from inexperienced students to 
staff doctors. The programs were also used by auxiliary medical professionals. (See Table 6.) 

The table reveals that over half the usage was logged on four programs. There is absolutely no 
relationship between the type of program and the type of user. This finding is puzzling in view of the 
assumed needs of different classes of users. However, this confirms experiences at other user institutions 
where many students wished to be exposed to the information although they had not yet acquired the 
requisite background for full comprehension. If the programs had been associated with specific courses 
the findings would probably have been different. 

MCV's curriculum development apparatus is an interdisciplinary committee. This committee 
chooses the topics to be taught and the objectives for each topic area. The classroom schedule has been 
reduced from 50 to 23 hours. One aspect of this curriculum reform movement has been the develop- 
ment of a number of self-study packages as a replacement for lectures. Dr. Young observed that 
"Everyone says that lectures are a poor method of instruction except the guy who's giving the lecture." 
It IS already clear from the CAI experience of the user institutions that these programs will not replace 
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Table 5 



Student Judgment of Supplier Institutions' CAI Programs 





Supplier Institution^ 


Student Judgment 


MGH 


UIMC 


osu 


Total 


1 learned terms, facts, classifications, theories, 
and procedures. 


1 


6 


10 


17 


1 learned to translate scientific knowledge into new 
concepts, interpret scientific data, and summarize 
material. 


1 


0 


3 


4 


1 learned to apply knowledge to problem situations. 


15 


7 


6 


28 


1 learned to analyze various elements or relationships 
in a problem. 


5 


1 


3 


9 


1 learned to develop hypotheses and design proposed 
plan(s) to solve a problem. 


2 


0 


1 


3 


1 learned to judge the efficacy of the proposed plan 
in solving the problem. 


3 


0 


1 


4 


Total 


27 


14 


24 


65 



•x2+ 24.58; p<. 01. 



Source: Wcrgin, 1973. 



lectures. This will happen only if there is a specific educational program such as the Independent Study 
Programs at the Universities of Washington, Wisconsin, and Ohio State. 

Dr. Young pointed out that students now being graduated are not self-sufficient. "They still need 
experience and guidance. Some areas of medicine cannot be taught well with machines . . . keep the 
artificiality out of it." 

Dr. Young also pointed out that students are now being overwhelmed by a variety of learning 
alternatives. "Some compulsive bright students explore all of these, but some are not able to, and get 
frustrated." This phenomenon was mentioned at visits to several other institutions. However, in all the 
student comments, there is no indication that they feel bogged down by the sheer weight of the 
available CAI materials. A few students did, however, comment that their study schedules were so tight 
that they could not avail themselves of the CAI materials as much as they would like. 

An advantage of CAI of particular importance to the medical field (where there is an avalanche of 
information) is that it facilitates retention through its question and answer approach. Not too many 
students actually read textbooks. "Most books present the whole field, and medical students cannot 
become experts in all fields," Dr. Young explained. 
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Table 6 

MGH Program Usage, by User Type 





User Typc^ 


Program 


Student 
(Pre- 
Cterkship) 


Student 
(Post- 
Clerkship) 


Intern 


Resident 


Post- 
Training 
and Faculty 


Other^ 


Total 
Times Used 


Abdominal Pain 


27 


37 


9 


6 


3 


2A 2N 
2D 2X 


90 


Coma 


25 


27 


12 


15 


3 


1A IN 
ID IX 


86 


Cardiopulmonary 
Resuscitation 


17 


21 


7 


8 




IP 3N 
ID IX 


59 


Diabetic Ketoacidosis 


16 


21 


8 


4 


2 


1A 
ID 


53 


Jaundice 


8 


13 


4 


5 


1 


1A 

ID IX 


34 


Pediatric Cough and 
Fever 


5 


12 


2 


2 


1 


IP 
ID 


24 
24 


Anticoagulant 
Simulator 


6 


6 


3 


2 




1A ID 
IP IX 


10 


Hypertensive 
Emergencies 


8 


7 


3 


1 




IN 
IX 


21 


Cardiac Simulator 


7 


6 


1 


2 


1 


IX 


18 


Idiopathic Respiratory 
ulsxress in ixewDorn 


5 


5 


1 


1 


2 




14 


Oigoxin Dosage 
Advisor 


3 


2 


1 






ID IX 
2P 


10 


Drugs in Renal Failure 


1 


2 








IP 


4 


Gl Bleed 


2 


1 










3 


Total 


130 


160 


51 


46 


13 


37 


437 



group of students studying together at the terminal is counted as an entity and receives a count of 1. 
^ A = Auxiliary medical professions 
D » Dentists 
N ° Nurses 
P = Pharmacists 
X = Nonmedical persons 

Source: Wergin, 1973. 
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chapter 4 

EVALUATION OF CAi LIBRARY MATERIALS 

The combined libraries of Massachusetts General Hospital, Ohio State University, 
and University of Illinois Medical College' provided a wide range of biomedical subjects 
and pedagogical approaches for users to sample. 

The program usage data for selected institutions are examined here. These institu-# 
tions and the reasons for their selection are: 

• UCLA— is largest user. 

• Stanford--has highly unstructured curriculum. 

• Oregon-includes School of Nursing users. 

• Medical College of Wisconsin-has special experimental course in gross 
anatomy using CAI. 

Evaluative comments from various institutions, as reported in the User Bimonthly 
Report*, are a part of this report. Those institutions include: 
Hantman, Bolt Beranek and Newman, Inc. 
Medical College of Virginia 

National Board of Medical Examiners, American Board of Internal Medicine 
University of California— Davis, Office of Medical Education 
University of Arizona, Arizona Medical Center 
Washington University, St. Louis, Mo. 
University of Pennsylvania 

State University of New York, Upstate Medical Center, Division of Educational 

Communications 
Northwestern University 

University of Texas, Health Science Center at San Antonio 
National Naval Medical Center, Naval Medical Research Institute 
Buffalo Children's Hospital, Division of Instructional Communications 
University of Nebraska 

PROGRAMS IN THE MGH LIBRARY^ 



The MGH programs are primarily simulated clinical encounters with patients. The 
target population is composed mainly of medical students in their clinical years of 
training. Nineteen different programs are available on a wide range of subjects. Some of 
the subject areas are: 
Arrhythmias 

Hypertensive Emergencies 
Abdominal Pain 

*UIMC left the Network in February 1974. Their progrtmi, CASE (trtniferred to OSU) and CRIB, 
are not included in thit analysis. 

^For a complete description of MGH programs, the reader is referred to the Medical Education 
Programs U$ert* Manual Uboratory of Computer Science, Massachusetts General Hospital (undated). 
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Coma 
GI Bleed 

Pediatric Cough and Fever 

Trauma 

Jaundice 

Joint Pain 

Diabetic Ketoacidosis 

MGH also provided programs on drug dosage: Digoxin Dosage Advisor, Drugs in Renal 
Failure, and Anticoagulant Simulator. 

Since the programs are capable of creating an endless number of patient situations 
with varying symptoms, disease progression, and prognoses, the number of possible 
patient encounters is unlimited. However, after a student has sampled 10 or more 
patients within a program area, he is probably ready to progress to another segment. 
Average time to complete an encounter ranges from 10 to 30 minutes, depending on 
the simulation. 

The primary purpose of the simulation programs is to help the students improve 
their clinical judgment. The simulation programs are a systematic attempt to define an 
explicit model of the intellectual processes involved in medical decisions. Under the 
traditional apprenticeship method of clinical training, the student receives increasingly less 
supervision and monitoring of his performance and may be unaware of weaknesses in his 
problem-solving skills. The clinical simulation programs are a means of self-assessment as 
well as a vehicle for certification or licensure. (Proceedings of the Twelfth Annual 
Conference on Research in Medical Education, 1973.) 

Faculty and; student response to the MGH library was generally favorable and often 
highly enthusiastic. Some MGH programs were far more popular among students than 
others, even though they did not stand out on faculty ratings. One reason suggested for 
this popularity is that these programs provide the student with simulated clinical experi- 
ence in treating cases they usually do not see in their "live" clinical experience. The four 
most popular MGH programs among clerkship students were: 

The Comatose Patient (COMA) 
^ Abdominal Pai^i (ABD) 

Diabetic Ketoacidosis (DKA) 

Cardiopulmonary Resuscitation (CPR) , 
Available information on usage, faculty opinion, evaluation, and user comments for three 
of these programs is provided on the following ps^es. For contrast, information is also 
provided on one less popular MGH program—Hypertensive Emergencies. User comments 
indicate this program is too specialized for widespread use. However, it may be valuable 
for special purposes. 
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COMA (The Comatose Patient) 



AUTHORS: Edward P. Hoffer and Harvey Botman 
Massachusetts General Hospital 

DESCRIPTION: This program presents a model for evaluating a comatose patient as such patients are 
presented to the emergency department. Thirty-eight causes of coma are considered, including 
both primary neurologic disorders and toxic-metabolic conditions. The user's goal within the 
program is to arrive quickly and efficiently at a differential diagnosis and to begin appropriate 
therapy. Work-up of the patient, then, must be directed toward differentiating among the most 
probable diagnoses and also toward gathering data needed to make immediate management 
decisions. 

The setting in each of the cases is an emergency department. The user is given a brief introductory 
description of the "patient," and can then request further history, physical findings, and lab test 
data by entering the appropriate number from the vocabulary list. Therapies can be ordered at any 
time, again by selecting numbers from the vocabulary. 

TARGET AUDIENCE: Medical students in clinical years and beyond. 

USAGE: COMA was one of the most popular MGH pj'ograms. It was used over a thousand times at 
UCLA from November 1972 to August 1974 (581 hours usage). 

FACULTY OPINION: COMA was reviewed and rated by faculty at SUNY Downstate Medical Center, 
Mayo Foundation, Albany Medical College, and Medical College of Virginia. Compared to other 
MGH simulated patient encounters, COMA did not receive unusually high ratings by faculty, on 
either medical content or pedagogy. 

COMMENTS: "Tough, really difficult, well thought out. Evidences a fantastic anrwunt of research." 

Report #4, CAPO Use of the Lister Hill Communications Network, Period 
Ending August 30, 1973, From Leonard M. Hantman, Bolt Beranek and 
Newman, Inc. 

"Good simulation and review; well explained, evaluation helpful; introductory part should be 
shorter; does not explain vi'hy certain lab tests should not be ordered." 

"Best structure; very good on difficult diagnosis and acute management." 

May 1973 Operational Users Report, Medical College of Virginia, David L Hopp, 
Ph.D., Assistant Professor, Educational Planning and Development Program, 
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ABDOMINAL PAIN 



AUTHOR: Octo Barnett, Massachusetts General Hospital 

OESCRIPTION: In this computer-based exercise, the student is given a short description of a patient 
with a chief complaint of abdominal pain. The patient may have any one of 33 different causes of 
acute abdominal pain. The student has available an index of history, physical exam, and lab data 
items about which he can obtain Information, in any order he desires. The quality of the student's 
performance is measured by the efficiency with which he sequentially selects questions or tests, 
and by the accuracy with which he interprets the information. The different items are considered 
to have different costs (e.g., a history question is less expensive than a laboratory test). 

TARGET AUDIENCE: Medical students in clinical years and beyond. 

USAGE: ABDOMINAL PAIN was one of the most popular MGH programs. It was used about 1,500 
times at UCLA during the period November 1972 to August 1974 (443 hours usage). 

FACULTY OPINION: Faculty who rated ABOOMINAL PAIN along with other simulated patient 
encounters, did not give it particularly high marks on content or pedagogy. 

- PROGRAM EVALUATION: "The results (of the evaluation) were most intriguing and indicated that 
the major Increase in the ability to make a proper diagnosis of a problem of abdominal pain came 
between the second and fourth year of medical school, and that relatively little gain in diagnostic 
ability was evident between the graduating medical students and the more experienced physicians. 
The latter group were clearly superior, however, to the graduating medical students as well as to 
the younger medical students with respect to the efficiency with which they reached the diagnosis, 
the speed with which they reached the diagnosis, and the reduction in costs associated with the 
overall workup process. These preliminary data provide some most intriguing insights into the way 
in which people learn diagnostic skills, and the stages at which certain aspects of these skills may 
be acquired." 

Letter from Johr) /?. Senior,, M.D,, Natior^al Board of Medical Examiners, 
American Board of internal Medicine, June 1, 1973. 

"Programs rated as most interesting . . .'were as follows: 

Coma: 9 responses 

Abdominal pain: 6 responses 
Cardiopulmonary 

resuscitation: 6 responses" 

Letter report from Richard F. Walters, Ph.D,, Assistant Professor, Office of 
Medical Education, University of California, Davis, May 16, 1973. 

COMMENTS: "The ABO Pain program was by far the most popular program among the student users. 
It was referred to as a great learning experience and challenge, the performance evaluation was 
good; it was fun and educational. The student users felt it would build their problem-solving 
ability . . . 

Overall, those student users that did fill out evaluation slips, were willing to put thought into their 
comments. The enthusiasm was high among the student users and they did much of the promotion 
for the program by word of mouth. Of course the users did not have prior terminal experience nor 
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instruction via computer, for an evaluative point of reference. Nonetheless, the present responses 
indicate the worth of the program." 



Arizona Medkal Center, CAI Report, October 15, 1973 - December 15, 1973., 
The University of Arizona. 

"Those listed by students represent basically problems in instructional format. They included: 

(1) inability to ask certain questions not in the vocabulary (e.g., in the abdominal pain program) 

(2) inability to start at any point in the program other than the beginning (3) inherent limitation 
in the programs to get or give information except that which has been previously specified by the 
author of the program." 

Summary Evaluation: MGH Computer Assisted Instruction Programs, from 
Linda Smith, Trainee in Computer Librarianship, Washington University, 
April 19, 1973. 

"Tht abdominal pain program, developed as a testing and not as a teaching program, is not 
suitable in its present formulation as a teaching program: the interpretation is too involved for 
student understanding, and timewise, it is too long for efficient use of student-computer 
contact time," 

Biomedical Communications Network Progress Report, no. 1, April 1973, from 
Jerome S. Rauch, Chief Medical Librarian, Project Director, University of 
Pennsylvania. 

"... one student stated that the statistics reported in the abdominal pain section were pointless." 

Letter report from Richard F. Walters, Ph.D., Assistant Professor, Office of 
Medical Education, University of California, Davis, May 16, 1973. 

"The abdominal pain problem ... is very popular. The detailed analysis of student errors, however, 
is considered tedious." 

Bimonthly Report to Educational Component Data Communications Service 
(LHC), from Herbert Schneiderman, M.D., Associate Director (CAI), Division of 
Educational Communications, Upstate Medical Center, State University of New 
York, March 15, 1973. 

"A brief description of a patient coming to an emergency ward is given to the student, who then, 
with the aid of a glossary, can elicit a history and do physical examination and lab tests, in order 
to make his diagnosis. An evaluation of the student's management is also presented upon request." 

Report on Usage of MGH-CAI by Julian Aguda, May 9, 1973, Northwestern 
University. 

"Program did not understand my diagnosis (noise on line, other transmission problems?) although 
they were identical to program generated answers." 
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"Trouble with machine typing extra characters as I was trying to type." 

"Garbage was occasionally received; program was cut off without warning." 

University of Texas Health Science Center at San Antonio, Analysis of CAI 
Utilization, Jan, 18 - Feb. 28, 1973, From David A. Kronick, Ph,D,, Librarian 
and Director, Medical Communications, 



50 



DIABETIC KETOACIDOSIS 



AUTHOR: Edward P. Hoffer, Massachusetts General Hospital 

DESCRIPTION: This program presents a dynamic simulation of a patient in diabetic coma. The user is 
given a brief initial description of the patient and may then request physical findings, order lab 
test$, and institute therapies in that order. The "patient" is represented by a mathematical model; 
this rriodel is updated based on considerations of time and user interventions, and the "patient's" 
condition changes accordingly. (Dr. Hoffer, in a recent paper, described the rationale for using 
computer simulation nnodules.) (Hoffer, 1973) The physical exam • lab test therapy cycle is 
repeated for a simulated hour at a time, until the "patient" either is brought into metabolic 
balance or dies. 

Important parameters controlled by the model include blood sugar and acetone, serum K, 
serum pH, urine volume, etc. The interaction seems to be realistic enough that the student receives 
a vivid exposure to the challenge of therapeutic management of a complex time-changing 
disease entity. 

An interactive program attempting to instruct in the methods of therapeutics and management 
using simulated models of diabetic patients. 

TARGET AUDIENCE: Medical students in clinical years and beyond. 

USAGE: DIABETIC KETOACIDOSIS was one of the most popular of the MGH programs. It was used 
about 600 times at UCLA in the period November 1972 to August 1974 (313 hours usage). 

FACULTY OPINION: Faculty members who rated this program in opinion surveys did not give it 
unusually favorable marks. 

COMMENTS: "[The student] found this program to be realistic, stimulating and a good review of a 
topic which nrwiny internists lose familiarity with, soon after leaving the residency and fellow- 



"I had no specific criticisms to make, except that the program did not permit me to obtain 
radiographs of the chest and abdomen, which I consider to be important diagnostic aids early in 
the management of this condition." 



ship years. 



Letter from Lawrence Raymond, M,D., Program Manager, Research Clerks, 
Naval Medical Research Institute, National Naval Medical Center, 
18 September 1973. 



Good introduction to patient management; feedback good; difficult to understand.' 



May 1973 Operational Users Report, Medical College of Virginia, David /. 
Hopp, Ph.D., Assistant Professor, Educational Planning and Development 
Program. 
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"The students appreciated the ability of several programs, particularly Diabetic Ketoacidosis, to 
delay laboratory results just as would happen in an actual case, and the student must proceed with 
his therapy prior to receiving all of his laboratory data." 

Lister Hill Center CAI Experimental Network Users Report, for the period 
January 'February 1974 by Ronald G. Davidson, M,D., Director, Division of 
Instructional Communications, Buffalo Children's Hospital, March, 1974, 

"One student cited the diabetic ketoacidosis as being too complicated." 

Letter report from Richard f. Walters, Ph.D., Assistant Professor, Office of 
Medical Education, University of California, Davis, May 16, 1973, 
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HYPERTENSIVE EMERGENCIES 



AUTHOR: Edwird P. Hoffer, Massachusetts General Hospital 

DESCRIPTION: This program presents a simulated hypertensive patient, and uses a series of multiple- 
choice questions to direct the user in managing the "patient." Occasionally the user Is asked to 
name a drug choice rather than select one from a list. Type the name of the drug to be prescribed; 
any reasonable drug selection will be recognized by either its generic or trade name. 

Each management decision is commented on, and the "patient's" reaction to the management is 
reported. The interaction continues until the "patient" is stabilized on therapy, or in some cases is 
lost to follow-up* 

TARGET AUDIENCE: Medical students in clinical years and beyond. 

USAGE: HYPERTENSIVE EMERGENCIES received relatively low usage. It was used about 120 times 
at UCLA in the period November 1972 to August 1974 (28 hours usage). 

FACULTY OPINION: Hypertensive Emergencies received slightly higher ratings than other MGH simula- 
tions by staff at Mayo whose opinions were solicited. 



"Material is good but too much emphasis on dosage and specific drugs for student level. These are 
rare problems, although important, but I believe more common things should be emphasized. The 
material is appropriate for HO 1 or 2. I think content is too specialized for our usual student in 
three year program." 



COMMENTS: "Well liked.' 



Report #A, CAPO Use of the Lister Hill Communications Network, Period Ending 
August 30, 1973, from Leonard M. Hantman, Bolt Beranek and Newman inc. 



Comments by R. i. Grissom, Summary Comments on the Evaluation of 
CAl Materials, University of Nebraska 
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PROGRAMS IN THE OSU LIBRARY^ 



The materials in the OSU CAI library are designed for use in the following 
major areas: 

(1) Curricula for medicine, nursing, and allied health professions, as a supple- 
mental teaching tool. 

(2) Continuing medical education for health professionals. 

(3) Student self-study and self-evaluation in an independent study environment 
for basic science for medical students. 

Ohio State has approximately 120 available instructional units offering approxi- 
mately 300 interactive hours of CAI. The CAI materials cover a wide range of subject 
matter and pedagogical techniques. Some of the course materials are: 

Reading the Patient's Medical Records 

Acid-Base Balance 

Enzyme: Identification, Classification, and Significance 
Gross Anatomy Self-Evaluation Exercise 
Hi, Baby (for new mothers) 
Histology 

Juvenile Diabetes-Patients 
Examination of the Fundus 
Oral Cancer Recognition 
A Bird's-Eye View of Pediatrics 
Cardiac Arrhythmias 
Anticoagulant Medication 
Differential Diagnosis of Toothache 

PILOT Independent Study Program (33 self-evaluation modules (encompassing 

the pre-clerkship portion of the medical school) 
Medical Terminology 

Most require the use of slides or other auxiliary materials at the terminal. The time to 
complete any one program ranges from a few minutes to several hours. 

Use patterns for OSU materials are far more variable than for the MGH materials. 
Use of the OSU materials is highly dependent on curriculum integration. Thus, a 
particular program may receive very high usage one semester because an instructor assigns 
it as part of the course work. The program may be dropped after the course is over. 

Similarly, the target audience for OSU materials varies widely. Thus, a program 
receiving very high usage in a nursing school may not be used at all in a medical school. 

Most OSU programs are targeted toward the pre-clinical years of medical training. 
Thus, audience use does not overlap greatly with the MGH library which is 
clinically oriented. 

Five OSU program packages receiving very high usage at one or more of the sample 
institutions are discussed on the following pages. 



* For a complete description of the OSU materials, the reader is referred to User*s Guide to Computer 
A$$iited InMtruction (CAl), Ohio State University College of Medicine, Septenibor 1974. 
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PSYMED (Psychotropic Medications) 



AUTHOR: Kathleen Lennon, RN 

DESCRIPTION: Major tranquilizers, minor tranquiliz-srs, and antidepressants are reviewed in this pro- 
gram. General information on these three classifications is coupled with specific material on the 
most commonly used medications. 

TARGET AUDIENCE: Nurses, physicians, pharmacy 

USAGE: PSYMED was the most used program by the University of Oregon where the School of Nursing 
was the heaviest user. (Total of 99 hours usage from February 1973 - September 1974.) Forty 
senior students in psychiatric nursing were assigned the use of PSYMED by their instructor. Since 
PSYMED takes approximately 2 1/2 hours to complete, these 30 students may account for ail of 
the usage. PSYMED received almost no use at Stanford, UCLA, or the Medical College 
of Wisconsin. 

FACULTY OPINION: No data 

PROGRAM EVALUATION: Forty senior students were assigned the program PSYMED dunng 
Quartern. The students were randomly assigned to two groups: one group (10 students) to 
complete the program individually, the second (30 students) to complete the program in randomly 
assigned groups each comprising three students. Within-and'between-course comparisons were to be 
made on the basis of mid-term and final examinations sampling the content identified in the 
program. The control group had been identified as those students in NUR 327 (Quarter I); 
experimental groups consisted of the two identified during Quarter II, and two additional groups 
to be identified during Quarter III. Unfortunately, technical difficulties were encountered and an 
insufficient number of students completed the program to make valid statistical comparisons. The 
original intention was to compare the students' performance on standard Board scores and the 
National League of Nursing scores. However, the only data produced were subjective. The nursing 
students who completed the CAI program reported that the program materially improved their 
knowledge level. 

Project Coordinators: Ms. Doris Schindler 

Assistant Pn-^fessor of Psychiatric Nursing 
University of Oregon 
School of Nursing 

Mr. Rick Duffleld 

Instructor of Psychiatric Nursing 

University of Oregon 

School of Nursing 
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HIST1 (HISTOLOGY) 



AUTHOR: Beth L. Wismar, Ph.D., Ohio State University 

DESCRIPTION: Utilizing a great number of slides, HIST1 aids the student in recognizing the identifying 
characteristics of 40 organs. The majority of the questions will be accompanied by two 
photomicrographs of the same organ but different magnifications. You will be asked to study both 
slides before identifying the organ. A few questions will present only a list of structures and 
tissues from an organ and you will be expected to identify the organ from the description. 

The organ ID course is a teaching device, not a test. Some of the organs shown may be new to 
you and you may not get them correct on the first try, but you should be able to "logic" then] 
out. In the context of this course, you will be asked to distinguish between fundic and pyloric 
stomach, cervix, uterus, various calibers of vessels and many of the organs of the body. 

TARGET AUDIENCE: Health professionals who desire a review in histology; first year medical students. 

USAGE: At the Medical College of Wisconsin, HIST1 received the second highest OSU usage in hours; at 
UCLA it received fifth highest usage in hours. It did not receive any usage at Stanford or Oregon. 

PROGRAM EVALUATION: "Students who had demonstrated a weakness In tissue and organ identifi- 
cation were advised to utilize the OSU HISTI program, in addition to continued work at the 
microscope. Most of the 16 students in this group did so. In previous practical examinations on 
tissue and organ identification, these 16 students scored 15% points below the class average (69.7% 
against a class average of 84.3%). On the final practical examination, this group of students scored 
78% against a class average of 78.9%. We feel that the OSU HISTI program contributed 
significantly to the astounding recovery demonstrated by this group of students." 

January - February 1974 Lister Hill Center User Report, Medical College of 
Wisconsin (MCW 222) from Weston D. Gardner, M.D., Professor of Anatomy, 
LHCBC Program Monitor. 

"The HISTI program was used on a voluntary basis by about 30 students in the General Pathology 
course of the Medical Program. In addition an equal number of students taking a histology course 
also used HISTI on a voluntary basis. The students in the histology course were both 1st year 
medical students and undergraduate pre-meds. 

I felt that HISTI is an excellent program to use in conjunction with the teaching of Pathology and 
Histology. The format Is excellent and the program has been cleverly put together. However I 
would hope that In the future more teaching can be Incorporated Into this type of program. While 
realizing the difficulties involved, it would appear that the best kind of program for organ 
Identification would be one In which a large number of comments, suggestions or explanations are 
presented before the organ Is actually Identified. The order In which the various organs were 
presented was not very obvious to us. It very likely reflects a sequence of teaching used 2t OSU 
and it would be useful information for trial users to receive when using the program." 

Evaluation of CAI Programs on TYMSHARE Network, from Nelson Fausto, M.D., 
Associate Professor, Medical Sciences (Pathology), Brown University, May 30, 1973. 

"There is no particular pattern to periods of demand for the system which are dependent upon 
the area of study by students who are scattered throughout the curriculum. Recently, students, in 
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reviewing earlier material, have gathered in groups of 4-7 to take various courses like 
HIST. . .They appear to have strong positive feelings about the educational value of this kind of 
interaction, which seems to approach class recitation." 

Lister Hill Center CAI Experimental Netvifork • Operational Users Report, from 
Ralph E. Cutler, M.D., Director, Washington Independent Study Program, 
University of Washington, 18 June 1973. 

"The technical unavailability of OSU complete HIST1 program, with continual access only to the 
DEMO program compromised the effectiveness of an announced program which proved to be 
largely unavailable to them. 

The inability to commence later study at the specific point to which a student had progressed 
previously compromised the effectiveness of programs, especially OSU HIST1. 

Although apparently approved, the inability of 'go to' shifts in program alternatives within d 
category reduced effectiveness. Students complained that if evaluation on a group of program 
items showed satisfactory knowledge they had to complete a whole unit before the vendor 
computer would allow a shift." 

January - February 1974 Lister Hill Center User Report, Medical College of 
Wisconsin (MCW222) from Weston D. Gardner, M.D., Professor of Anatomy, 
LHNBC Program Monitor. 
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GANAT1 (Gross Anatomy Self-Evaluation Exercise-tutorial) 

AUTHORS: Medical students and faculty of the Anatomy Department, Ohio State University 

DESCRIPTION: Review and self-evaluation in anatomy. Ten sections, each with a multiple choice test, 
include: anterior abdominal wall; abdomen; pelvis, thora, lower and upper extremities; back, head, 
neck, eye. 

TARGET AUDIENCE: Any health professional as well as first year medical students. 

USAGE: GANAT1 was the most used (in hours) of ail the OSU programs used at UCLA, and at the 
Medical College of Wisconsin. It was used only an hour or two at Stanford and at Oregon. 

FACULTY OPINION: GANAT1 was given a low rating on pedagogy by faculty at SUNY Downstate 
Medical Center. It received low ratings by Mayo staff on "flexibility of response to student error" 
and "quality of feedback." Quality or validity of program content was given an average rating by 
both groups. 

PROGRAM EVALUATION: The Medical College of Wisconsin used the Ohio State University Gross 
Anatomy Course, GANAT1, in a paired experiment in teaching Gross Anatomy. In the 1973-74 
academic year, the Department of Anatomy presented two simultaneous courses in medical 
Gross Anatomy. 

Ninety-seven students formed a base group undertaking Gross Anatomy in the conventional course 
(Section A) which has evolved from other course development in the past. 

Section B was a special course. For this course 24 students were selected to represent a cross 
section of abilities in the class. The basic selection process was to take every fifth student from a 
ranked list, but derandomizing the selection process to include at least one black and one female). 
Section B, directed by Dr. Weston D. Gardner, undertook a seminar and peer group instruction 
approach, using less scheduled classroom time. This group explored body systems in a series of 
tutorial seminars. Laboratory work consisted of in-depth study of one major body region per 
' student. The latter part of the course was devoted to peer group instruction, in which students, 
who had made the in-depth study of a particular region, teamed up to use a variety of audio-visual 
methods to explain their understanding of the region to the other students. A particular feature of 
the Section B approach was the use of CAI and self-evaluation, using programs from OSU, UIMC, 
and MGH to self-evaluate their progress in learning. 

The Section B course was a practical attempt to learn if a shortened systems course, with (a) less 
individual dissection but increased tutorial-seminar discussions, (b) varied audio-visual reinforce- 
ment, (c) computer-assisted self-evaluation, and (d) peer group instruction could be considered a 
valid approach in modern curricular planning. 

Both Section A and Section B students were administered extra-mural final examinations (prepared 
under the auspices of the Association of Anatomy Chairmen): The National Board of Medical 
Examination and a short answer fill-in, diagram-type more objective examination. 

"Preliminary results on the National Board Examination show that while there was no significant 
statistical difference between the conventional and special sections in the first year of the study 
(1972-73), this year's special section achieved a marked and significantly higher performance. The 
final report will present numerical figures and comparative data between the 1971-72 base group, 
and the conventional and special groups in 1972-73, and 1973-74. 
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"Although it appears that the availability of CAI programs in 1973-74 may be mainly responsible 
for the higher achievement in the special section, several other variables must be and are 
being evaluated."^ 

COMMENTS: Use by students in this period shifted away from 0SU-HIST1 and GANAT2 to 
0SU-GANAT1 and the 8 modules of OSU-PILOT for which special arrangements were made. 
Despite the fact that the OSU staff tend to deprecate their GANAT1 (as the oldest program with 
fewer program options such as branching) GANAT1 remains the single most popular most used 
program available to our students. Our program, despite the distinct asset of the addition of the 
PILOT modules, could not have been the success it has been (see Section V) without the 
evaluation»interaction»instruction enhancement derived fromGANATI. 

Lister Hill Center User Report, Medical College of Wisconsin, March 1974 



*This paragraph and one preceding are quoted commenw from Weston D, Gardner, "User Report/' January/ 
Ftbruary 1974. 
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ACIBA (Acid-base Balance) 

AUTHORS: Bob Clark, Medical Student, Ohio State University 
Thomas Williams, M.D., Ph.D., Ohio State University 

DESCRIPTION: Contains basic information on the following topics: respiration and blood-gas transport; 
quantitative acid-base balance (pH calculations); graphic analyses of acid-base balance; compensa- 
tion systems (renal, pulmonary, body cells), anion gaps. 

TARGET AUDIENCE: Clinical lab personnel, medical students, nurses, physicians, and student nurses. 

USAGE: ACIBA received very high usage at nearly all of our sample schools. Its popularity is indicated 
by the following rankings in OSU program use for the period February 1973 - August 1974: 

Oregon 

(Nursing) UCLA Stanford Wisconsin M.C. Virginia 



3rd highest 3rd 2nd low use 3rd highest 

use highest highest (4 hrs) 

FACULTY OPINION: ACIBA was given average ratings by Mayo staff who evaluated it. 

PROGRAM EVALUATION: NUR 312-Clinical Experience in Medical-Surgical Nursing 

Sixteen sophomore students assigned to one clinical rotation will be assigned to review the 
programs ACIBA, COMA, JAUNDICE, VEINS, and STROKA (assigned for both 311 and 312; 
each of the 16 students, however, will review the program once). Performance of this rotation will 
be compared with the performance of the remaining NUR 312 rotations (14-16 students in each 
rotation) on criteria Including the quality of nursing care plans produced, clinical performance, and 
test performance in NUR 311. 

Project Coordinator: Ms. Donna Schantz 

Assistant Professor of Medical-Surgical Nursing 



50 



6U 



PILOT 



AUTHORS: Staff, Ohio State University *" 

DESCRIPTION: The individual modular computer programs within the Ohio Stale University College of 
Medicine Independent Study Program serve as self evaluation exercises for students who have 
studied a prescribed body of information. The ISP encompasses the preclerkship portion of 
medical school and the units of study are organized by body systems. Each modular self*evaluation 
(tutorial evaluation system, TES) presupposes student preparation in the unit of study (module) 
and an understanding of the designated modular instructional objectives. The questions within each 
self-evaluation exercise are not intended to cover all the specific modular instructional objectives, 
but rather to sample the student's knowledge of selected areas within the designated unit of study. 
The Independent Study Program is an integrated program of study with inter-disciplinary input to 
each module of study. Discipline coverage of a particular subject within a module can be obtained 
through examination of the instructional objectives. Therefore, the module objectives are not 
always to be considered as total coverage of a given subject, nor is the TES designed to measure 
the student's total comprehension of the instructional objectives. 

TARGET AUDIENCE: First two years of medical school; health personnel 

USAGE: At UCLA, PILOT was not used until September 1974. For the two months of September and 
October 1974, PILOT modules were used a total of 22 hours at UCLA, making PILOT one of the 
most popular OSU programs at the time. 

PILOT was used far more than any other OSU program at Stanford-a total of 122 hours from 
January to October 1974. 

At both UCLA and Stanford, students pursue independent paths of study. The PILOT programs 
are ideally suited to student needs in these environments. 



51 




Chapter 5 
COSTS AND BENEFITS 



COSTS 

The Lister Hill costs for the first two years of Network operation (July 1, 
1972- July 1, 1974) were approximately $847,000-$344,000 for TYMSHARE, the com- 
munications contractor, and $503,000 for the Network contractors. 

The TYMSHARE cost is subdivided into fixed costs (those costs that do not vary 
with increased usage) and variable costs. The fixed costs include the rental of TYCOM 
(there is a special communications computer at each host site), maintenance of the user 
name file, and invoice preparation. The TYCOM rentals are $1,950 per month, and the 
administrative overhead (user names and invoice preparation) a little over $1,000. 

The variable portion of the TYMSHARE cost is broken down into connect time and 
characters transmitted. Variable costs are $5.43 per instruction hour. In a typical month's 
usage of 1800 hours, the variable cost would be $9,774. 

The contractor costs are divided into two parts: part 1, the charge for the computer 
costs, part 2, the chaa^e for the personnel support. Part 1 costs are based on a 
$600/month cost per communications port. Contractors are paid $600 per port each 
month. The number of ports for which Lister Hill has paid has varied, but as of 
September 1974, Lister Hill was paying for 14 ports— 7 at each contractor center (OSU 
has donated two additional ports). At this rate of port usage, the annual hardware costs 
are $100,800; the annual personnel support costs are approximately $101,200. 

Table 7 shows the total CAI cost per terminal hour. The table assumes there are 
1800 terminal hours per month usage. (Actual monthly usage has varied from 1500 to 
2300 hours.) 



Table 7 

Total CAI Costs Per Terminal Hour^ 



Component 


Cost (Per Terminal Hour)^ 


TYMSHARE variable communication cost 


$ 5.43 


TYMSHARE fixed communication cost 




(TYCOMS. user names, invoice preparation) 


3.28 


Computer port charges 


4.66 


Computer host personnel support costs 


4.69 


NLM Central Staff 


1.66 


Total cost per terminal hour^ 


$19.72 



^Does not include user institution costs for terminals, personnel, materials, 
or local communications facilities. 

terminal hours are not always the same as student instruction hours. 
Students may work together in smaH groups. 
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Effective cost for a student hour of instruction is sometimes much lower than the 
terminal-hour cost. Students commonly work in small groups of two to four when using 
the MGH clinical simulations. For a group of three students, then, the cost per student 
hour would be a little over $6.00. 



COSTS IN RELATION TO BENEFITS 

One finding that emerged clearly from the Network Experiment was that the 
benefits of CAI distribution accrued directly to students but not to the institutions or 
faculty. Students benefitted in a number of ways, receiving: 

• Remedial learning. 

• Counseling. 

• Review for examinations. 

• Ehroblem-solving experiences. 

• Computer awareness they can apply later as practicing physicians. 

• Wide range of simulated clinical experience. 

None of these learning experiences replaced existing methods. Therefore, the costs 
do not displace any existing costs of medical training. The costs, and the benefits, were 
in addition to the existing curriculum. Consequently, it is inappropriate to compare the 
costs of the CAI learning experience with the costs of other learning modalities, such as 
lecture, small-group conference, individual counseling, or making clinical rounds. Such 
comparisons imply that the CAI might displace these other costs. Experience, to date, has 
not supported this. 

The question is whether tlie addition?d benefits to students are worth the cost to the 
institutions or to the National Library of Medicine. General consensus seems to be that a 
cost of $19+ per hour is too high, even when maximum benefit is assumed. 

Could the CAI resource benefit the institution? At OSU, there have been no cost 
savings by going to an independent study program for the first two years of medical 
training. The number of students could theoretically be increased without much increase 
in costs for the final two years. However, the clinical portion of training (the third and 
fourth years) would have to be increased to handle the additional students. Faculty- 
student ratio and hospital facilities required for clinical years of training are regarded as 
fixed constraints. 

One other possible benefit to the institution would be the increased quality of its 
products as measured by performance of their graduates in Medical Board examinations. 
There are indications that such a benefit could be demonstrated. However, the complex, 
controlled studies needed to show such benefit have not been made. 
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Chapter 6 

WHAT WAS LEARNED FROM THE NETWORK EXPERIMENT 



Harold Wooster, Special Assistant for Program Development, Lister Hill National 
Center, presented a list of questions he hoped to have answered (Wooster, 1973). The 
summary findings are organized around these questions. 

Use . "If CAI services are made available over a time-sharing communi- 
cations network, will people be willing to sample them?" 
The monthly average total time usage of approximately 2,000 hours attests to the 
fact that not only did they sample the materials but continued to use them repeatedly. 
Curricula . **Can the material offered over the Network be integrated 
in a meaningful and useful way into existing curricula? Can it 
produce measurable benefits? Will it produce changes in the ways 
medical schools teach?" 
The user institutions did not integrate the curriculum materials into their main-line 
curricula. Opinion surveys indicate that the MGH materials were perceived by both 
faculty and students to be supplementary learning devices. The OSU materials, on the 
other hand, were integrated to some degree. The faculty at a number of institutions 
assigned specific CAI learning modules as part of their classwork. While basic integration 
of the materials into the curriculum did not take place (except in isolated instances), this 
was not an obstacle to a high level of usage. 

Many problems arose in attempting to measure specific educational benefits. 
Initially, the institutions envisioned a comprehensive program of evaluation for the CAI 
materials, but most did not accomplish their original objectives. Faculty found that they 
could not commit enough time to actually carry out a controlled study. 

The Medical College of Wisconsin was a notable exception, since it conducted a 
scientific study of the effects of the Ohio State ISP anatomy modules. 

The Medical College of Virginia encountered several difficulties in evaluating the 
cost-effectiveness of the network at the time it was deciding whether to continue. These 
difficulties included the following (Wergin, 1973): 

• The usage statistics were not wholly comparable (e.g., data from Ohio State 
University and University of Illinois Medical College did not break down the 
usage by type of user). 

• User resistance to answering questionnaires was encountered. 

• Usage data were not related to the curricula (course performance). 

• It was not possible to relate opinion survey data to cost/benefit 
decision-making. 

The introduction of the Network into the schools* did not bring about noticeable 
changes in the institutions' teaching approach. The one outstanding case in which an 
entirely new educational program was affected, was the University of Washington's 
adoption of Ohio State's entire independent study program: philosophy, learning 
objectives, and program modules. (Although they modified the learning objectives and the 
CAI modules, the basic thrust of the program remained the same.) The Washington, 
Alaska, Montana, and Idaho (WAMI) satellite program, also operated by the University of 
Washington, is using the interactive satellite communication capabilities to bring the CAI 
programs to remote areas in these states. 
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CAI Materials Development . "What is the mechanism whereby new 
CAI material may be generated most efficiently, and cost effec- 
tively?" 

This question was not answered by the Network Experiment. While Lister Hill 
initially expected the users to contribute materials to the Network CAI library as a 
condition of participation, this did not materialize. The predominate number of faculty 
members at user institutions did not have the proper incentives to spend time learning 
the prerequisite authoring skills, nor did they have time to do the actual authoring. Many 
of the faculty are preoccupied with research work in addition to their teaching responsi- 
bilities and, thus, were not interested in becoming CAI authors. Others who were 
interested in teaching felt that the present reward structure would work against them. 

The contractors, per se, did not have a budget allocation for training user faculty as 
CAI authors. However, they did have a general fund for user support, primarily for 
support services. 

Dr. Octo Bamett of Massachusetts General Hospital felt that the best use of 
available resources could be made through in-house authoring of materials since it would 
be time consuming to work with a number of authors from different institutions. Dr. 
Bamett has recently modified his view; he is now more receptive to working with other 
institutions on the development of materials. 

Ohio State University, on a selective basis, has encouraged outside authorship. Dr. 
Paul Rehkopf of the University of Pittsburgh School of Medicine, Eye and Ear Hospital, 
is currently developing a series of programs on emergency room eye care. These simula- 
tion programs deal with the patient's medical history, physical examination procedures, 
and results of lab tests. The program content is being written at the University of 
Pittsburgh, and the computer programming is performed at Ohio State University by the 
instructional programming staff. The University of the Pacific is developing a series of 
cases to add to the CASE program library. The cases were developed online, using the 
CASE system driver program. 

It was found that revision of existing materials is far more cost-effective than 
beginning over again on the same type of program. Washington Independent Study 
Program (WISP) faculty members, spending on the average only 2 hours per month over a 
2-year period, have been able to make substantial revisions on Ohio State's Independent 
Study Program (ISP) whose initial development took many years and several 
million dollars. 

Data Base Gaps . "What is the mechanism whereby gaps in the data 
can be identified and filled?" 
The Network did not provide a direct answer to this question. However, some 
individual suggestions were made for areas and subjects in which there were clear data 
gaps. Users found that programs were needed which simulate nursing assessment (i.e., the 
identification of a problem that requires the immediate attention of a physician). These 
programs would be similar to the MGH programs on the clinical diagnostic simulation and 
treatment. Emergency medicine was an area that was frequently mentioned as a pro- 
ductive area for development of new CAI materials. 

Many faculty members made the observation that the current Lister Hill library was 
limited. The Network Experiment pointed out the need for a comprehensive survey of 
users to identify the most important areas for library CAI expansion. This survey could 
be subdivided by class of user. 

Network Organization . "How should the system be organized and 
managed? What is the best way to ensure interaction between users 
and providers so that the desires of both will be heard and 
acted on?" 

The main weakness of the Lister Hill Network organization was that it did not 
involve key decision makers in the user institutions. This made it difficult to develop the 
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proper level of support necessary to make the Network self-sustaining. Perhaps some type 
of policy board would have proved beneficial. The development of an effective human 
network proved to be the best mechanism for interaction between users and providers. 
Ftce-to-face contact between users and providers resulted in noticeable shifts in attitudes 
and policy. 

One mechanism for assuring interaction was the on-line comments feature. This 
provoked thousands of comments from individual CAT library users. Another feature not 
available in the Lister Hill Network that would have added to this interaction is a 
teleconferencing facility in which two or more parties may carry on a conversation (e.g., 
faculty CAI authors from two different institutions conduct a conversation with a 
Network host about revision and expansion of the library's emergency medi- 
cine materials). 

Site visits by host institution faculty with their counterparts in the user institutions 
were a definite stimulus to those curricula changes that occurred in the Network. After a 
site visit by MGH, the University of Pennsylvania offered an introductory course in 
clinical medicine. 

Distribution . "What is the most cost-effective distribution (net- 
working) mechanism?" 
The Network Experiment did not provide an answer to this question. Certainly the 
Network, as an experimental breadboard, was not the most efficient form of communica- 
tions network. Communications costs were nearly 50% of total Network costs of 
$19/terminal-use hour. The remainder of the costs was for computer usage, host support 
personnel, and National Library of Medicine central staff costs. 

If Lister Hill wishes to obtain concrete data on the most cost-effective distribution 
methods, it will have to conduct experiments on different distribution methods. This 
would involve producing cost estimates of these methods (e.g., the cost of a distributed 
computer network). In order to determine the actual cost, it would be necessary to 
measure the personnel cost component in an operational environment. 

Economics . "How can the Network be made economically viable— 
that is, how can it be placed on at least a partially self-sustaining 
basis?" 

The Network Experiment has proven that CAI distribution can be partly self- 
sustaining. Network users are currently paying almost the entire variable component of 
the Network communications cost, which amounts to over 50% of the total communica- 
tions cost. For example, the total bill to TYMSHARE for July 1974 was $12,959.91 of 
which $7,859.90 was billed to the user institutions. The question still remains whether 
the Network can be made fully self-sustaining. A stripped-down version of the present 
Network would cost between $8-12 per instruction hour. The $8 is a minimum amount 
which would cover only hardware costs, while the $12 figure would allow for some 
personnel support. 
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chapter 7 

FUNCTIONAL REQUIREMENTS OF A CAI DISTRIBUTION SYSTEM 

Certain functions related to distribution of CAI materials proved to be required in 
conjunction with the hard-wired Network. 

(1) User Tailoring. The Ohio State's CAI tutorials demanded considerable user 
tailoring for acceptance at the individual schools. At the University of Washington 
every ISP received some revision. At the request of user institutions, Ohio State 
developed mechanisms for "individualizing" the learning units to meet user institu- 
tions* specifications. 

(2) Feedback-Revision Process . One of the most valuable characteristics of the 
national Network was the ability to collect feedback on the material from users at many 
institutions. Users of the MGH clinical simulation programs wrote comments online as 
they went through a program. This information on content and pedagogy was based on 
immediate day-to-day learning experiences of students and faculty. Using these com- 
ments, authors ^of the materials at MGH were able to make continual refinement and 
improvements to the programs. (On page 58 are some examples that a user wrote as he 

went through a program.) 

(3) Support Services , A key requirement of any distribution system is support 
services to facilitate user adoption and use. Ohio State had one full-time staff member 
whose primary function was to answer user questions. Users inquired about such things as 
the availability of particular learning materials, the availability of visual materials 
accompanying programs, the feasibility of writing their own CAI modules, the objectives 
and suitability of learning materials for their own student body, problems encountered in 
accessing the host institution, and programming enors encountered in some learn- 
ing modules. 

In addition to such services at the host institution, there was a considerable 
need for the same kind of service at local user sites. Learning laboratory directors and 
librarians were available to help overcome the initial hurdle of using a computer system. 
They answered questions on how to log onto the system, what steps tc take in case of an 
accidental disconnect, and generally, to assist and orient people. 

(4) User Documentation . In the initial stage of the experimental Network, there was 
a lack of instructions on how to use the CAI library, resulting in a great deal of 
aggravation and frustration. The contractors, in response to this problem, developed 
comprehensive user guides (Ohio State University College of Medicine, 1974; Massa- 
chusetts General Hospital, Laboratory of Computer Science, undated). User institutions 
also developed local instruction sheets and booklets to help fulfill the informa- 
tion requirements. 

(5) Peer Review . The Network Experiment indicated that the absence of a review 
body was a significant deterrent to the enlargement of a CAI library. In the absence of a 
review board, the host institutions (contractors) had to pass judgment on whether new 
CAI materials developed at other institutions should be accepted for Network national 
distribution. Faculty members at user institutions asked that their materials be added to 
the Network library, but they were not permitted to do so. The contractors were 
concerned about the reputation of their institution and were, therefore, reluctant to 
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USE OF HYPERTENSIVE EflERGENCIES 



flARTlNHORN (53.30,271 .40, flSH) 'GEO HASH U l' 10/05/74 

STUDENT, HAVE NOT BEGUN PRINCIPAL MEDICAL CLERKSHIP 
10/05/74 7:38 Pfl TOT: 0.0S 
GOOD PROGRAM 



USE OF GI BLEED 



JOHN MAHAN (7.01, 2.43, JDM) 'HAHNEMANN' 10/05/74 

STUDENT, HAVE BEGUN OR HAVE COMPLETED PRINCIPAL MEDICAL CLERKSHIP 
10/04/74 9. -49 PM TOT: 0.08 

HOW WOULD YOU KNOW THAT THE PAT. BRUISED EASILY AND HAD 
NNOSEBLEEDS IF HE WAS UNRESPONSIVE, ESP. SINCE THE DX IN THIS CASE 
SEEMS ESP. DEPENDENT ON THE HX' GIVEN BY PAT. (AND KNOWING THAT 
IN THE MIDDLE OF NIGHT A PTT AND PT ARE HARD TO COME BY,) 



JOHN MAHAN (7.01,2.43, JDM) 'HAHNEMANN' 10/05/74 

STUDENT, HAVE BEGUN OR HAVE COMPLETED PRINCIPAL MEDICAL CLERKSHIP 
10/04/74 10. -21 PM TOT: 0.]0 

HOW WAS ACUTE GASTRITIS RULED OUT WITHOUT HAVING THE RESULT 
OF THE ENDOSCOPY KNOWN? 



LEVI YASGUR (7.01, 2.44,LHY) "HAHNEMANN" 10/05/74 

STUDENT, HAVE NOT BEGUN PRINCIPAL MEDICAL CLERKSHIP 
10/05/74 11. '29 AM TOT: 0.02 

HOW TO DIFFERENTIATE THIS FROM GASTRIC ULCER/ 



RAMON DOMINGO (53.30,271 .39, RKD) 'GEO WASH U I" 10/05/74 
MEDICAL SUB-SPECIALITY, FELLOW, 3 TO 5 YEARS OUT OF MED SCHOOL 
■ 10/05/74 5:50 PM TOT: 0.0] 

WHYDIDYOUNOTALLOW ME TO GASTROSCOPE THE PATIENT / 



USE OF ARRHYTHMIAS 



JOE MAKINEN (53.30,271 .38, LJM) "GEO WASH U 1' 10/04/74 

STUDENT, HAVE NOT BEGUN PRINCIPAL MEDICAL CLERKSHIP 
10/04/74 5:30 PM TOT: 0.05 
H,X= 125, 1=217,05 
FIGURE 9 DOES NOT LOOK LUKE WENCKEBACH. 
HX=125, 1=222 

DEAR SIR TRANSVENOUS PACING (1 ) COSTS MONEY (2) CAUSES 
PATIENNT DISCOMFORT AND (3) NECESSITATES AN INVASIVE 
PROCEDURE WHICH COULD CONCIEVABLY LEAD TO DANGEROUS 
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include materiak in the library which they felt were of questionable ment. Ruann E. 
Pengor and Dr. Robert Beran of Ohio State have repeatedly stressed the need for an 
outside review body to pass on the quality of CAI learning materials. 

(6) People Networks . In the early months of the Network Experiment, there was 
Uttle or no collaboration among user institutions. The Network program did not provide 
explicit mechanisms for user cooperation. 

Later, it became apparent to the users that a "people network was needed in 
conjunction with the hard-wired network. The Health Education Network Users' Group 
(HENUG) was formed in the latter part of 1974 to satisfy this need. HENUG was formed 
under the auspices of the Association for Development of Computer-Based Instructional 

Systems (ADCIS). ^ ^. ^. ^ . 

Article II of the HENUG by-laws spells out the functions which the members 
believe the "people network" needs to perform. Article II, Objects of the Society, is 
as follows: 



ARTICLE II 



Objects of the Society 

Section 1. To maintain and enhance medical education through inter-institutional sharing 
of materials as initiated by the National Library of Medicine's Expenmental 
CAI network. 

Section 2 To investigate academic and professional aspects of computer-aided instruction 
(including but not Umited to) faculty development, curriculum development, 
authorship, evaluation, and peer review. 

Section 3 To investigate technical and functional aspects of computer-aided instruction 
(including, but not limited to) networking alternatives, program transfer, docu- 
mentation, distribution, and cost effectiveness. 

Section 4. To develop computer-aided instruction into a disciplined literature. 

Sections. To communicate with users and with medical educators in general on the 
status, availability, and effectiveness of computer-aided instruction. 

Section 6. To provide meetings for the reading and discussion of papers on computer- 
aided instruction, to discuss policy, and to disseminate knowledge. 

(HENUG, 1974) 
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Chapter 8 
PROGRAM RECOMMENDATIONS 



Experimentation with the Lister Hill Biomedical Network pointed out the need for 
further research in a number of directions. 

(1) Research on Efficient Distribution Systems. Costs continue as a significant 
barrier to the use of CAI as a major teaching modality. The Library should consider 
sponsoring a long-term research program on different methods of CAI program distribu- 
tion. The Lister Hill Center should sponsor experimental programs using different delivery 
system methods. The Network Experiment has shown that relatively few CAI programs 
account for the bulk of CAI usage. If these programs were distributed in a standard 
language or languages to libraries having compatible intelligent terminals, the cost could 
be compared with other means of distribution. A report has been prepared for the Lister 
Hill Center on programming languages used in health sciences CAI programs (Brigham, 
1973). This report will be helpful in considering the problems and prospects of lan- 
guage standardization. 

Because of the high cost of national networking, the Library should establish 
experimental programs which try out other approaches to distribution, such as mini- 
computers, desktop computers, and intelligent terminals. A report has been prepared for 
the Lister Hill Center on minicomputers, comparing them to large computers 
(Brigham, 1973). 

(2) Research on the Instructional Benefits of CAI Programs. The Lister Hill 
Network Experiment indicates a strong need for more concrete evidence of the beneficial 
nature of the CAI programs. In-depth studies of the learning benefits are called for. The 
Library, therefore, should sponsor a continuing evaluation program. 

(3) Survey of CAI Library Needs. The Lister Hill Experiment indicates the inade- 
quacy of the scope of the Library programs. A survey of the educational needs of various 
user populations should be undertaken to learn those subject areas in which CAI 
materials are lacking. 

(4) Development of a n On-Line Bibliography for CAI Programs . The availability of a 
CAI program bibliography service will facilitate the exchange of CAI programs. Learning 
resource medical center personnel and biomedical librarians need information on learning 
materials available in specific subject areas and compatible v^rith their computer environ- 
ment. The availability of a catalog of health sciences CAI programs will facilitate the 
establishment of an on-line bibliography (Brigham, etai, 1973). 

(5) Development of a CAI Review Board. A recurring problem in the experimental 
network was the lack of a constituted body to judge the merits of CAI programs. 
Network hosts were placed in the position of having to determine the quality of their 
own programs. It would be of significant benefit to involve the various specialty boards 
and associations in a review function. The American Association of Medical Colleges 
would be the logical organization to develop the overall objectives, goals, and procedures 
for establishing CAI program review mechanisms. 

(6) Establish an Experimental Clearin^ouse for CAI Programs . Many CAI pro- 
grams developed at local computer centers are not available for general distribution. 
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This means that each institution requesting a CAI program must individually negotiate 
the release of that program. There are numerous legal and logistical problems 
associated with the release of CAI programs for general distributior . 

Ohio State encountered difficulty in working out the copyri^t arrangements 
for the visual display accompanying many of the CAI programs. An experimental 
clearinghouse function would test out many of the problems associated with exchange 
and distribution. 
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Appendix A 

INSTITUTIONAL AFFILIATES IN THE NETWORK EXPERIMENT 



USER INSTITUTIONS: 

Alabtnrn, Univ. of, School of Medicine 
Charles W. Slack, Ph.D. 
Director, CASAM Project 
University of Alabama in Birmingham 
Birmingham, Alabama 35233 

Albany Medical College of Union University 

Thomas C. Gabriele 

Director, Compiiter Science Center 
Albany Madical College 
Albany, New York 12208 

American Academy of Orthopedic Surgeons 
Henry J. Mankin, M.D. 
Chief of Orthopadics 

American Academy of Orthopedic Surgeons 
Massachusetts General Hospital 
Boston, Massachusetts 02114 

Richard M. Carter, Ph.D. 

Educational Consultant 

American Academy of Orthopedic Surgeons 

Chicago, Illinois 60611 

Tom Nelson 

Assistant Executive Director 

American Academy of Orthopedic Surgeons 

Chicago, Illinois 60611 

American College of Radiology 
Roger Bauman, M.D. 
Chairman, Department of Radiology 
American College of Radiology 
Massachusetts General Hospital 
Boston, Massachusetts 02114 

American Heart Association 
Thomas K. Burnap, M.D. 
Boston Hospital for Women 
American Heart Association 
Brookline, Massachusetts 02146 

American Thoracic Society 
Barnard Pennock, Ph.D. 
Univ. of Oklahoma Health Sciences Center 
American Thoracic Society 
Oklahoma City, Oklahonw 73190 

Robert M. Rogers, M.D. 

Chairman, Medical Education Committee 

American Thoracic Socie^ / 

New York, New York 10019 



Arizona, University of 
Daniel A. Hill 

Medical Audio-Visual Services 
University of Arizona Medical Center 
Tucson, Arizona 85724 

Richard A. McNeely 
Instructional Media Specialist 
Medical Audio-Visual Services 
University of Arizona 
Tucson, Arizona 85724 

William Harrison 
Assistant Director 
Medical Audio-Visual Services 
University of Arizona 
Tucson, Arizona 85724 

Susan E. Miller 
Instructional Media Specialist 
Medical Audio-Visual Services 
University of Arizona 
Tucson, Arizona 85724 

James W. Berry 

Librarian, University of Arizona 
Tucson, Arizona 85724 

Augusta General Hospital 
Henry J. Wheelwright, M.D. 
Medical Director 
Augusta General Hospital 
Augusta, Maine 04330 

Beth Israel Hospital 
Howard Bleich, M.D. 
Beth Israel Hospital 
Boston, Massachusetts 02215 

Bird, Kenneth T., M.D.-Massachusetts General Hospital 
Kenneth T. Bird, M.D. 
Massachusetts General Hospital 
Boston, Massachusetts 02114 

Boston University Medical Center, University Hospital 
T.M. Teager, Sc.D. 

Boston University Medical Center Hospital 
Boston, Massachusetts 021 1 8 

Brown University 
Nelson Fausto, 

Assistant Professor of Pathology 
Division of Biological & Medical Sciences 
Brown University 
Providence, Rhode Island 02912 
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Buffalo Children't Hospita) (Continuing Education, MGH) 
Ronald G. Davidion, M.D. 
Proffssor and Associate Chairman, 
Department of Pediatrics, and 
Director, Division of Instructional Communications 
Buffalo Chiidran's Hospital 
Buffalo, New York 14222 

Miss Marcia Chelminiak 
Buffalo Children's Hospital 
Buffalo. New York 14222 

CAPO (Bolt, Beranak and Newman, IncJ 
Leonard M. Hantman 
Medical Systems Engineer 
Bolt, Beranak and Newman, Inc. 4^ 
Cambridge. Massachusetts 02138 

Bruca Bradley 

Matthew Thornton Health Plan 
Nashua, New Hampshire 03060 

William W. Bookheim. P.A. 
Matthew Thornton Health Plan 
Nashua, New Hampshire 03060 

California, Univ. of— Davis 
Richard F. Walters, Ph.D. 
Associate Professor, Office of Medical Education 
University of California at Davis 
School of Medicine 
Davis, California 95616 

California, Univ. of-Los Angeles 
Robert E. Robinson, III, M.D. 
Assistant Director 
Hospital and Clinics 
Medical Center 

University of California at Los Angeles 
Los Angeles, California 90024 

Sandi Parker 

Library. The Center for the Health Sciences 
Medical Center 

University of California at Los Angeles 
Los Angeles, California 90024 

Bernice M. Wenzel. Ph.D. 
Assistant Dean for Educational Research 
University of California at Los Angeles 
Los Angeles, California 90024 

Louise Darling 

Head Librarian. Biomedical Library 
Medical Center 

University of California at Los Angeles 
Los Angeles, California 90024 

Gloria Werner 

Assistant Biomedical Librarian 
Medical Center 

University of California at Los Angeles 
Los Angeles, California 90024 



California, Univ. of-Los Angeles, Cedars-Sinai Medical Center 
Lois Ann Colaianni 
Director of Libraries 
Cedars-Sinai Medical Center 
University of California at Los Angeles 
Los Angeles. California 90054 

California, Univ. of-Los Angeles, Harbor General Hospital 
Carl W. Trygstad 

Course Director, Third Year Medical Students 
Harbor General Hospital 
University of California at Los Angeles 
Los Angeles. California 

Eleanor Goodchild 

Medical Librarian 

Harbor General Hospital 

University of California at Los Angeles 

Los Angeles, California 

California. Univ. of-San Diego 
E. Roger Marchand, Ph.D. 
Office of Learning Resources 
University of California at San Diego 
La Jolla. California 92037 

John Sewell 
Supervisor 

Learning Resource Center 
University of California at San Diego 
La Jolla. California 32037 

California. Univ. of-San Diego Hospital 
Lawrence J. Saidman. M.D. 
Professor and Chairman 
Department of Anesthesia 
University of California Hospital, San Diego 
San Diego, California 92103 

Richard Uhl, M.D. 

Department of Anesthesia 

University of California Hospital, San Diego 

San Diego, California 92103 

Case Western Reserve University 
Peter G. Katona, M.D. 

Associate Professor of Biomedical Engineering 
Case Western Reserve University 
Cleveland. Ohio 44106 

David M. Waggoner. M.D. 
Director, Primary Health Clinic 
Cleveland Metropolitan General Hospital 
Case Western Reserve University 
Cleveland, Ohio 44109 

William Stickley 

Health Sciences Communications Center 
Case Western Reserve University 
Cleveland, Ohio 44106 
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CBX Projtct, National Board of Medical Examiners 
John R. Senior, M.D. 
CBX Project 
3930 Chestnut Street 
Philadelphia, Pennsylvania 19104 

Jack D. Myers, M.D. 
Acting Chairman 
CBX Test Committee 
Philedalphie, Pennsylvenia 

Donald A.B. Undbery, M.D. 
Chairmen 

CBX Advisory Group 
Philadelphia, Pennsylvenie 

Chicago, University of 
UW. Coppleson, M.D. 
University of Chicago Hospitals & Clinics 
Chicago, I ll'mois 60637 

Cincinnati, University of 
Xharles Schwenkner, M.D. 
Department of Biomedical Communications 
University of Cincinnati 
Cincinnati, Ohio 45219 

Columbia University, New York-College of 
Physicians end Surgeons, Medical Library 
J.T. Bigger, M.D. 

Associate Profnsor of Medicine end Pharmacology 

Columbia University 

New York, New York 10032 

Norman Kahn, D.D.S., Ph.D. 
Associate ProfMSor of PharmKology 
Columbia University 
New York, New York 10032 

Merril Schindler 

Columbie University Medical Library 
New York, New York 10032 

Columbie University, New York-College of 
Physiciens and Surgeons, Department of Pathology 
Kenneth F. Button, M.D. 
Department of Pathology 
College of Physiciens & Surgeons 
Columbie University 
New York, New York 10032 

Paul Krieger 

Department of Pathology 
College of Physicians & Surgeons 
Columbie University 
New York, New York 10032 

Connecticut, University of 
Robert G.Votew, Ph.D. 
Director, Muttidisciplinary Labs 
University of Connecticut Health Center 
Far mington, Connecticut 06032 



Emergency Medical Systems Medical Group 
Gregory B. Camp, M.D. 
Suite 405 

Town & Country Village 
Palo Alto, California 94301 

Florida, Univ. of-College of Dentistry 

Arthur D. King, Ph.D. 
Instructional Systems Coordinator 
College of Dentistry 
University of Florida 
Gainesville, Florida 32601 

Florida, Univ. of-Department of Medicine 
Lamar Crevasse, M.D. 
Department of Medicine 
J. Hillis Miller Health Center 
University of Florida 
Gainesville, Florida 32601 

C. Benjamin Stevens 
Assistant in Medical Education 
Department of Medicine 
University of Florida 
Gainesville, Florida 32601 

Ft. Worth (Texas) Osteopathic Hospital 
George M. Esselman, D.O., F.S.C.0.1. 
Director of Medical Education 
Fort Worth Osteopathic Hospitel 
Fort Worth, Texas 76107 

Jeanne A. Gilday 

Project Monitor, CAI 

Fort Worth Osteopathic Hospital 

Fort Worth, Texas 76107 

George Washington University 
Charles S. Tidball, Ph.D., M.D. 
Office of Computer-Assisted Education 
The George Washington University Medical Center 
Washington, D.C. 20037 

Georgia, Medical College of 
E.Jay Wheeler, M.D., Ph.D. 
Director, Medical Curriculum Development Program 
Medical College of Georgia 
Augusta, Georgia 30902 

Curtis H. Carter, M.D. 
Dean, School of Medicine 
Medical College of Georgia 
Augusta, Georgia 30902 

John J. Dallman, Ph.D. 
Supervisor, Computer Operations 
Medical College of Georgia 
Augusta, Georgia 30902 

Hahnemann Medical College of Philadelphia 



Robert F. Johnston, M.D. 
Professor of Medicine and Director, 
Division of Pulmonary Diseases 
Hahnemann Medical College & Hospital 
Philadelphia, Pennsylvania 19102 

lit- 




Hahnemann M<dic<l College of Philadelphia ( Con t ) 
Daniel W. Shirley, B.S. 
Division of Pulmonary Diseases 
Hahnermnn Medical College & Hospital 
Philadelphia, Pennsylvania 19102 

Harvard Medical School-Francis A. Countway 
Library of Medicine 
Harold Bloomquist 
Librarian 

Francis A. Countway Library of Medicine 
Harvard Medical School 
Boston, Massachusetts 021 15 

Marcia Ford 
Systems Librarian 

Francis A. Countway Library of Medicine 
Harvard Medical School 
Boston, Massachusetts 02115 

Thomas K. Burnap, M.D. 
Assistant Professor of Anesthesia 
Harvard Medical School 
Boston, Massachusetts 02115 

Howard University 

David B. Henson, Ph.D. 

Assistant Professor of Biochemistry 

Department of Biochemistry 

Howard University College of Medicine 

Washington, D.C. 20001 

M.Willy, M.D. 

Department of Medical Education 
Howard University College of Medicine 
Washington, D.C. 20001 

Illinois, Univ. of, Rockford-School of Medicine 
James Magero 
Director of Media Services 
University of Illinois, Rockford 
School of Medicine 
Rockford, Illinois 61101 

Illinois, Univ. of. Urbane— School of 
Basic Medical Sciences 
George L. Hody, M.D. 
School of Basic Science 
University of Illinois 
Urbane, Illinois 61801 

Johns Hopkins University 
Richard A. Polacsck, M.D. 
Director/ Librarian 
Welch Medical Library 
Johns Hopkins University 
Baltimore, Maryland 21205 

Robert S. Gresehover 
Welch Medical Library 
Johns Hopkins University 
Baltimore, Maryland 21205 



Kentucky, University of 
Timothy A. Smith, Ph.D. 
Director of Computer-Assisted Instruction 
College of Dentistry 
University of Kentucky 
Lexington, Kentucky 40506 

Dr. Garth Old 

Division of Computer Services 
University of Kentucky Medical Center 
Lexington, Kentucky 40506 

Walter W. Stoll, Jr., M.D. 
Associate Director 
Clinical Associate Program 
University of Kentucky 
Lexington, Kentucky 40506 

Lome Linda University 

Dr. Ralph Korpman, M.D. 

Assistant Instructor and Computer Coordinator 

Department of Pathology 

Laboratory of Medicine 

Loma Linda University 

Loma Linda, California 92352 

Daniel P. Reeves 

Coordinator of Computer Assisted Instruction 

School of Allied Health Sciences 

Loma Linda University 

Loma Linda, California 92352 

Los Angeles County/USC Medical Center 
Stephen Karas, M.D. 
Department of Emergency Medicine 
Los Angeles County/USC Medical Center 
Los Angeles, California 90033 

Los Angeles County Medical Association 

Ben Z. Krentzman, M.D. 
Program Chairman, Computer Committee 
Los Angelas County Medical Association 
Santa Monica, California 90404 

Milton G. Sheinbein, M.D. 
Computer Committee Chairman 
Los Angeles County Medical Association 
Santa Monica, California 90404 

Louisville, University of 

Col. Thomas M. Murray, Jr. 

CAI Coordinator, Health Sciences Center 

University of Louisville 

Louisville, Kentucky 40201 

Maryland, University of 
Thomas Held 
Director of Media 
Office of Medical Education 
University of Maryland 
Baltimore, Maryland 21201 
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Mryland, University of (ContJ 

Albtrt L. Goldbtrg, Ed.D. 
Associate Director, Office of Medical Education 
Univeriity of Maryland School of Medicine 
Battimore« Maryland 21201 

Massachusetts Genera! Hospital-Continuing Education 
Edward P. Hoffer, M.D. 
Laboratory of Computer Science 
Dtptrtment of Continuing Education 
Massachuittts General Hospital 
Boston, Massachusetts 02114 

Stephen E. Goldfiner, M.D. 
Department of Continuing Education 
Massachusetts General Hospital 
Boston, Massachusetts 02114 

Massachusetts, University of 
William J. Cooke, Ph.D. 
Assistant Professor 
Department of Pharmacology 
University of Massachusetts Medical School 
Worcester, Massachusetts 01605 

Leslie Lipworth, M.D. 

Associate Profeuor 

Department of Community Medicine 

University of Massachusetts Medical School 

Worcester, Massachusetts 01605 

Mayo Foundation 

Merlin M. Mitchell, Ed.D. 

Director, Instructional Development Center 

Mayo Foundation 

Rochester, Minnesota 55901 

G.W. Beeler, Ph.D. 
Mayo Foundation 
Rochester, Minnesota 55901 

J. Daub, M.D. 
Mayo Foundation 
Rochester, Minnesota 55901 

McGill University 
Josephine M. Cassie 

Assistant to the Dean, Faculty of Medicine 

McGill University 

Montreal 109, Quebec, Canada 

Michigan State University 
Edward G. Titus, Ph.D. 
Office of Curriculum Implementation 
College of Human Medicine 
Michigan State University 
East Lansing, Michigan 48824 

Michigan, Univ. of— School of Nursing 
Samuel Schultz, II, Ph.D. 
Associate Professor of Psychology in Nursing 
School of Nursing 
University of Michigan 
Ann Arbor, Michigan 48104 



Michigan, Univ. of-School of Dentistry 
William F.Fitzgerald, Ph.D. 
Director, CAI DENT Center 
University of Michigan Dental School 
Ann Arbor, Michigan 48104 

Michigan, Univ. of-School of Medicine 
Gordon L. Nordby, Ph.D. 
M5416 Medical Science I 
University of Michigan 
Ann Arbor, Michigan 48104 

National Institutes of Health, Clinical Center 
Roderick E. Prior, M.D. 
Clinical Center 
National Institutes of Health 
Bethesda, Maryland 20014 

Thomas L. Lewis, M.D. 
Clinical Center 
National Institutes of Health 
Bethesda, Maryland 20014 

National Medical Research Institute, 
National Naval Medical Center 
M.L. McCreary, LCDR MC USN 
National Naval Medical Center 
Bethesda, Maryland 20014 

Lawrence Raymond, M.D. 
Program Manager, Research Clerks 
National Naval Medical Center 
Bethesda, Maryland 20014 

Nebraska, Univ. of-Medical Center 
Gerald J. Fleischii, M.D. 
Coordinator of Health Information Technology 
University of Nebraska 
Omaha, Nebraska 68105 

New England Deaconess Hospital 
Joseph L. Andrews, Jr., M.D. 
Chief, Pulmonary Laboratory 
New England Deaconess Hospital 
Boston, Massachusetts 02015 

New Jersey, College of Medicine and Dentistry 
Philip Rosenstein 
Director of Libraries 
College of Medicine & Dentistry 
of New Jersey 
Newark, New Jersey 07103 

Janet Crager 

Information Systems Specialist 
College of Medicine & Dentistry 
of New Jersey Library 
Newark, New Jersey ^7103 

Isabel Spiegel 

Information Systems Supervisor 
College of Medicine & Dentistry 
of New Jersey Library 
Newark, New Jersey 07103 
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New York, State Univ. of--Buf faio 
C.K. Huang 
Health Science Library 
State University of New York 
Buffalo, New York 14214 

Amy Lyons 
Health Science Library 
State University of New York 
Buffalo, New York 14214 

New York, State Univ. of-Children's 
Hospital, Buffalo (see also Buff. 
Chil. Hosp.) 

Ronald G. Davidson, M.D. 

Children's Hospital 

State University of New York 

Buffalo, New York 14222 

New York, State Univ. of— Downstate Medical Center 
Herbert S. Diamond, M.D. 
Office of Continuing Education 
State University of New York 
Brooklyn, New York 11203 

New York. State Univ. of-Stony Brook 



Gerald W. Miller, Ph.D. 

Associate Professor of Educational Research 

and Medical Psychology 
University of Oregon Medical School 

and School of Nursing 
Portland, Oregon 79201 

Walter A. Petersen 
Director, UOMS Computer Center 
University of Oregon Medical School 
Portland, Oregon 79201 

Pacific, University of the 
William G. Harless, Ph.D. 
Professor, Medical Sciences 
Pacific Medical Center 
University of the Pacific 
San Francisco, California 94120 

Marcia A. Zier 

ABIM MERIT Project Associate and 

Research Associate 
Pacific Medical Center 
University of the Pacific 
San Francisco, California 94120 



Alvin Bicker, M.D. 
Office of the Dean 
Health Sciences Center 
State University of New York 
Stony Brook, New York 11790 

New York, State Univ. of— Upstate Medical Center 
Herbert Schneiderman, M.D. 
State University of New York 
Syracuse, New York 13210 

Northwestern University 
George Czerlinski, Ph.D. 
Department of Biochemistry 
Northwestern University Medical School 
Chicago, Illinois 6061 1 

Will'am B. Buckingham, M.D. 
Associate Professor of Medicine and 
Director of Skills Laboratory 
Department of Medicine 
Northwestern University Medical School 
Chicago, Illinois 60611 

Eri Dordal, M.D. 

Northwestern University Medical School 
Chicago, Illinois 60611 

David Cugell, M.D. 

Northwestern University Medical School 
Chicago, Illinois 60611 

Oregon, Univ. of-Medical School and 
School of Nursing 
Dr. M. Roberts Grover, Associate Dean 
University of Oregon Medical School 
Portland, Oregon 79201 



Bruce E.Spivey, M.D. 

Dean of Medical Sciences 

Chairman, Department of Ophthalmology 

University of the Pacific 

San Francisco, California 94120 

Pennsylvania, Medical College of 
Claire Schultz 
Library 

Medical College of Pennsylvania 
Philadelphia, Pennsylvania 19129 

Pennsylvania, Univ. of-School of Medicine 
Jerome S. Rauch 
Chief Medical Librarian 
University of Pennsylvania School of Medicine 
Philadelphia, Pennsylvania 19104 

Mary G. Lawler 
Assistant Librarian 

University of Pennsylvania School of Medicine 
Philadelphia, Pennsylvania 19104 

Pennsylvania, Univ. of-Graduate Hospital 
Joseph A. Fortuna, M.D. 

Center for the Study of Enr^ergency Health Services 
Department of Community Medicine 
Graduate Hospital, University of Pennsylvania 
Philadelphia, Pennsylvania 19174 

Pittsburgh, Univ. of-School of Medicine 
(CBX Project) 
Jack Myers, M.D. 
Department of Medicine 
University of Pittsburgh School of Medicine 
Pittsburgh, Pennsylvania 15213 
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Pittsborgh, Univ. of-School of Medicme, 
Eye and Ear HotpiUl 
Paul Rahkopf 
Eye & Ear Hospital 
Univtrsity of Pittsburgh School 
of Mtdicina 
Pittsburgh, Ptnnsylvania 15213 

Dick Katzin, M.D. 
Dirfctor of Mtdical Education 
Eya & Ear Hospital 
Univtrsity of Pittsburgh School 
of Mtdicina 
Pittsburgh, Ptnnsylvtnia 15213 

Paul M. Grant, M.D. 
Dtpartmtnt of Ophthalmology 
Eya & Ear Hospital 
Univtrsity of Pittsburgh School 
of Mtdicina 

Pittsburgh, Ptnnsylvania 15213 

RAND Corporation 

Marshall Rockwtll, M.D. 
Tht RAND Corporation 
Santa Monica, California 90406 

Rochtsttr, Univtrsity of 
Paul L. Grovtr, Ph.D. 
Assistant Proftssor 
Division of Medical Education & Communication 
Univtrsity of Rochtster School of 
Mtdicina & Dtntistry 
Rochtstar, New York 14642 

Rush Medical College 

Gerald L. Gottlieb, Ph.D. 

Director, Office of Computer-Assisted Instruction 
Rush Medical College 
Chicago, Illinois 60612 

Rutgers University 

Anthony F. Barthotomay, M.D. 
Department of Community Medicine 
Rutgers Medical School 
. Piscatawty, New Jersey 08854 

St. Joseph and Children's Hospital of Orange County 
Barnard Kahn, M.D. 
St. Joseph and Children's Hospital of 

Orange County 
Orange, California 92668 

Helen Ashar 

Librarian, Burlaw Medical Library 
St. Joseph & Children's Hospital of 

Orange County 
Orange, California 92668 

South Carolina, Medical University of 
Joseph Mangiaracina 
Physiology Data Processing Center 
Medical University of South Carolina 
Charleston, South Carolina 29401 
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Sidney Katz 
Associate Professor and 

Director. Physiology Data Processing Facility 
Medical University of South Carolina 
Charleston. South Carolina 29401 

Karen A. Duncan. Ph.D. 

Assistant Professor 

College of Dental Medicine 

Office of Education 

Medical University of South Carolina 

Charleston. South Carolina 29401 

Southern California, University of 
Nelson J. Gilman 
Librarian, Norris Medical Library 
University of Southern California. School 
of Medicine 

Los Angeles. California. 90033 

Roger W.Jelliffe. M.D. 
Associate Professor of Medicine 
University of Southern California, School 
of Medicine 

Los Angeles, California 90033 

Elaine Adarm, Ph.D. 
Learning Resource Coordinator 
University of Southern California, School 
of Medicine 

Los Angeles. California 90033 

Southern Illinois University 
D. Dax Taylor, M.D. 
Associate Dean for Medical Education 
Southern Illinois University , School 

of Medicine 
Springfield, Illinois 62708 

Beth Dawson 

Department of Medical Education 
Southern Illinois University 
Springfield. Illinois 62708 

Reed Williams 

Department of Medical Education 
Southern Illinois University 
Springfield, Illinois 62708 

Marilyn Margon 
Staff Development Specialist 
Department of Educational Resources 
and Development 
Southern Illinois University 
Springfield. Illinois 62708 

Ron Comer 

Department of Medical Education 
Southern Illinois University 
Springfield, Illinois 62708 

Stanford University-^School of Medicine 
James B. Williams 

Director, Division of Instructional Media 
Stanford University School of Medicine 
Stanford, California 94305 
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Stanford Univ»rnty-School of Mtdicine (Cont) 

RMnattt Burlingamt 
FItischrmn Ttaching Ctnttr 
Stanford Univtrtity School of Medicina 
Stanford, California 94305 

John P, Sttward, M.D. 

Dtan of Studtnt Affairs 

Stanford Univtrtity School of Mt dicint 

Stanford. California 94305 

Taxi Collage of Ostaopi^thic ^4ad^cine 
Marion E. Coy, D.O. 
Prttidant 

Taxat Colltga of Ottaopathic Medicine 
Fort Worth, Taxai 76107 

Joan Swain 
Library Director 

Texas College of Osteopathic Medicine 
Fort Worth, Texas 76107 

Texas Tech University 
James E. Dyson, Ph.D. 
Assistant Dean for Education 
Texas Tech University School of Medicine 
Lubbock, Texas 79409 

Btair Rowley, Ph.D. 

Texas Tech University School of Medicine 
Lubbock, Texas 79409 

Texas, Univ. of- Sin Antonio 
David A. Kronick, Ph.D. 
Librarian 

University of Texas Medical School 
San Antonio, Texas 78229 

James Waldron, Ph.D. 
Director of Educational Resources 
University of Texas Medical School 
San Antonio, Texas 78229 

Texas, Univ. of-Galveston 

Edward N. Brandt, Jr., M.D., Ph.D. 
Associate Dean for Clinical Affairs 
University of Texas Medical Branch 
Galveston, Texas 77550 

Texas, Univ. of— Houston 
John J. Lenahan, Ph.D. 
Biomedical Communications 
Texas Medical Center 
University of Texas Medical School 
Houston, Texas 77025 

Texas, Univ of-Southwestern Medical School, Dallas 
David J. Mishelevich, M.D., Ph.D. 
Director, Medical Computing Resources Center 
University of Texas Southwestern Medical School 
Dallas. Texas 75235 



Tufts University-New England Medical Center 
Edward P. Leavitt 
Medical'Oental Librarian 
Tufts University 
Boston, Massachusetts 02111 

Margaret Young 

Tufts University Medical Library 
Boston, Massachusetts 02111 

Lauo F.Cavazos, Ph.D. 

Acting Dean 

Tufts University 

Boston, Massachusetts 02111 

Steven G. Pauker, M.D. 

Assistant Professor of Medicine 

TuftS'New England Medical Center Hospital 

Tufts University 

Boston, Massachusetts 02111 

Jerome P. Kassirer, M.D. 

TuftS'New England Medical Center Hospital 

Tufts University 

Boston, Massachusetts 021 1 1 

Utah, University of 

Mitchell Schorow, Ph.D. 
Director, Medical Sciences Education 
University of Utah Medical Center 
Salt Lake City, Utah 84112 

Priscilla M. Mayden 

Director, Eccles Medical Sciences Library 

University of Utah 

Salt Lake City, Utah 84112 

Vanderbilt University 
A.B. Brill, M.D. 

Director, Division of Nuclear Medicine 

& Biophysics 
Vanderbilt University Hospital 
Nashville, Tennessee 37232 

Virginia Commonwealth Univ.-Medical College 
of Virginia 
David Hopp, Ph.D. 
Assistant Professor 
Virginia Commonwealth University 
Medical College of Virginia 
Richmond, Virginia 23298 

John Wergin, Ph.D. 

Educational Planning & Development Program 
Virginia Commonwealth University 
Medical College of Virginia 
Richmond, Virginia 23298 

Loren WillianK, Ph.D. 

Director, Educational Planning & Development Program 
Virginia Commonwealth University 
Medical College of Virginia 
Richn>ond, Virginia 23298 
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Virynia Commonwealth Univ.-Medical ColleQc 
of Virflinia (Cont) 

Gloria H. * .jrwitz 

AisUtant Professor and Coordinator 
Learning Resource Facilities 
Virginia Commonwealth University 
Medical College of Virginia 
Richmond, Virginia 23298 

WHhinqton Hospital Center 
Tredtrick P. Ut¥«n, M.D. 
Washington Hospital Center 
W^hington, D.C. 20010 

Jay Shapiro, M.D. 
Waahington Hospital Center 
Washington, D.C. 20010 

Washington University "St. Louis 

Judith Lauter 
Librarian 

Washington University School of Medicine 
St. Louis, Missouri 631 10 

Barbara Halbroofc 
Reference Librarian, Library " 
Washington University School of Medicine 
St. Louis, Missouri 631 10 

Washington University 
Ga V E. Striker, M.D. 
Assistant Dean for Curriculum 
University of Washington 
% Seattle, Washington 96105 

Ralph Cutler, M.D. 

University of Washington School of Medicine 
Seattle, Washington 96105 

Charles W. Dohner, Ph.D. 

Director, Office of Research in Medical Education 

University of Washington 

Seattle, Washington 98105 

Thomas E. Morgan, M.D. 
Associate Dean for Academic Affairs 
University of Washington 
Seattle, Washington 98105 

Wayne State University 
Irwin K. Rosenberg, M.D. 
Department of Surgery 
Wayne State University School of Medicine 
Detroit, Michigan 48201 



Harry Maisel, M.D. 

Wayne State University School of Medicine 
Detroit, Michigan 48201 

Joseph W. Hess, M.D. 

Director, Division of Educational Services & Research 
Wayne State University 
Detroit, Michigan 48201 

West Virginia University 
Robert L. Murphy 
Director 

Medical Center Library 
West Virginia University 
Morgantown, West Virginia 26506 

Robert F. Munn 

Program Monitor and Director of Libraries 
West Virginia University • 
Morgantown, West Virginia 26506 

Wisconsi n. Medical College of 
F. David Anderson, Ph.D. 
ChairnDdn, Department of Anatomy 
Medical College of Wisconsin 
Milwaukee, Wisconsin 53233 

Weston D. Gardner, M.D. 
Professor of Anatomy 
Medical College of Wisconsin 
Milwaukee, Wisconsin 53233 

Sidney Shindell, M.D* 
Chairman, Computer Advisory Committee 
Medical College of Wisconsin 
Milwaukee, Wisconsin 53233 

Alfred A. Rimm, Ph.D. 
Chairman, Department of Biostatistics 
Medical College of Wisconsin 
Milwaukee, Wisconsin 53233 

Wisconsin, University of 
Donald R. Korst, M.D. 
Director, Independent Study Program 
University of Wisconsin Medical School 
Madison, Wisconsin 53706 

Thomas C, Meyer, M.D. 

Chairman, Department of Postgraduate Medical Education 
Univ. of Wisconsin Extension 
Madison, Wisconsin 53706 
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CONTRACTORS: 



Massachusetts General Hospital 

G. Octo Barnett, M.D. 
Director^ Laboratory of Computer Science 
Massachusetts General Hospital 
Boston, Massachusetts 02114 

Barbara B. Farquhar 
Assistant Director 
Laboratory of Computer Science 
Massachusetts General Hospital 
Boston, Massachusetts 02114 



Ohio State University 

Robert L. Beran, Ph.D. 

Associate Director 

Independent Study Program 

Ohio State University College of Medicine 

Columbus, Ohio 43210 

Ruann E. Pengov 

Director, Division of Computing Services 
for Medical Education and Research 
Ohio State University College of Medicine 
Columbus, Ohio 43210 



LISTER HILL CENTER: 
Jinnet Lewis 

Institute for Communication Research 

Stanford University 

Stanford, California 94305 

(Ms. Lewis was formerly with Lister Hill) 

Becky Lyon 

Technical Information Specialist 
National Library of Medicine 
8600 Rockville Pike 
Bethesda, Maryland 20014 

Harold Wooster, Ph.D. 

Special Assistant for Program Development 

National Library of Medicine 

8600 Rockville Pike 

Bethesda, Maryland 20014 
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NETWORK SERVICES NODE ACCESS 
FOR TYMCOMJII USERS 



Sf tt 

Arizona 

Belgium 

California 

California 

California 

California 

California 

California 

California 

California 

California 

California 

California 

California 

California 

California 

California 

California 

California 

California 

Colorado 

Connecticut 

Connecticut 

DC 
DC 

Florida 
Florida 
France 
France 
France 

Georgia 

Illinois 

Illinois 

Illinois 

Illinois 

Louisiana 

Louisiana 



City 

Phoenix 
Brussels 
Cupertino 
Cupertino 
Inglewood 
Inglewood 
Los Angeles 
Los Angeles 
Mountain View 
Newport Beach 
Oakland 
Oxnard 
Palo Alto 
Palo Alto 
Riverside 
Sacramento 
San Diego 
San Francisco 

San Jose 

San Jose 

Denver 

Darien 

Hartford 

Washington 

Washington 

St. Petersburg 

Tampa 

Paris 

Paris 

Paris 

Atlanta 

Chicago 

Chicago 

Freeport 
Freeport 
Baton Rouge 
New Orleans 



State 

Maryland 
Massachusetts 
Massachusetts 
Massachusetts 
Michigan 
Minnesota 
Missouri 
Missouri 
New Jersey 
New Jersey 
New Jersey 
New York 
New York 
New York 
New York 
New York 
New York 
Ohio 

Oklahoma 
Oregon 
Pennsylvania 
Pennsylvania 

Texas 
Texas 
Texas 
Texas 
Texas 
Texas 

Washington 
Wisconsin 



City 

Baltimore 

Boston 

Boston 

Cambridge 

Detroit 

Minneapolis 

Kansas City 

St. Louis 

Englewood Cliffs 

Union 

Wayne 

Buffalo 

New Yoi4< 

New York 

New .York 

Rochester 

Syracuse 

Columbus 
Oklahoma City 
Portland 
Philadelphia 
Pittsburgh 

Dallas 

Houston 

Houston 

Houston 

Midland 

San Antonio 

Seattle 

Milwaukee 
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